Read file spring batch

The next part of my Spring Batch tutorial describes how you can read the input data of your batch job from a relational database. You can get the example application of this blog post from Github. Leave a Reply Cancel reply. Home About Me Blog Tutorials. After you have read this blog post, you: Can identify the dependencies which are required when you want to read the input data of your batch job from an XML file.

Can get the required dependencies with Maven and Gradle. Understand how you can configure an ItemReader bean which reads the input data of your batch job from an XML file. Let's start by taking a quick look at the example application. This blog post assumes that: You are familiar with Spring Batch You can get the required dependencies with Maven or Gradle. You must configure the location of your input file by creating a new ClassPathResource object because I wanted to create an example application that's as easy to run as possible.

Typically, the input file of your batch job is found from the file system. Update the value of sourceBucket and sourceObjectPrefix based on the S3 bucket details of your use case. For this example, I have created a sample movie data set. Flat file item reader config to parse CSV file as below. For item processor, we will configure with async capabilities.

This is not needed for the sample data set for the example in this post. However, when you want to process a huge data set async processing will be very helpful. Then, for item writer we configure it with async.

This is to support the item processor which process the items in multiple threads and we need to handle the Future which gets returned. I have created two movie data set CSV files for this example. The data set files are available under resources folder in the project for reference. The execution logs are as below. In this article, we have explored how to read files from S3 bucket using spring batch. The key note is we have processed the files as stream without loading them in memory.

The complete source code is available over on Git. Chunk-oriented processing allows jobs to implement efficiently the most common batch processing tasks: reading, processing, and writing. We focus here on the first step of this process, reading. We describe general concepts and types implemented by Spring Batch.

These built-in types are the foundation used to support the most common use cases. Spring Batch can use different data sources as input to batch processes.

Because Spring Batch is open source, implementing and extending core types for reading is easily achievable.