If we specify the starting position in EventHub conf like so:
EventHubsConf(ConnectionStringBuilder(eventHubConnectionString).build)
.setStartingPosition(EventPosition.fromStartOfStream)
or
.setStartingPosition(EventPosition.fromEndOfStream)
And also sepecify the checkpoint location in the StreamWriter
streamingInputDF
.writeStream
.option("checkpointLocation", checkpointLocation)
...
After a restart, does the setStartingPosition
become irrelevant because the checkpoint is always used as the point from where to begin reading?
Thanks.
Read more here: https://stackoverflow.com/questions/66269887/azure-event-hubs-streaming-does-checkpointing-override-setstartingposition
Content Attribution
This content was originally published by Gadam at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.