Spring Batch read step in a loop
I came across a piece of code that reads some data like this:
public class StudioReader implements ItemReader<List<Studio>> {
@Setter private AreaDao areaDao;
@Getter @Setter private BatchContext context;
private HopsService hopsService = new HopsService();
@Override
public List<Studio> read() throws Exception {
List<Studio> list = hopsService.getStudioHops();
if (!isEmpty(list)) {
for (Studio studio : list) {
log.info("Studio being read: {}", studio.getCode());
List areaList = areaDao.getArea(studio
.getCode());
if (areaList.size() > 0) {
studio.setArea((String) areaList.get(0));
log.info("Area {1} is fetched for studio {2}", areaList.get(0), studio.getCode());
}
this.getContext().setReadCount(1);
}
}
return list;
}
However, when I start the job, this read is done in a loop. I found from another stackoverflow answer that this is the expected behavior. Then my question is the best solution for the given example? Extend StudioReader from JdbcCursorItemReader? I found one example that defines everything in the xml, which I don't want. And here is the context.xml part for the reader:
<bean class="org.springframework.batch.core.scope.StepScope" />
<bean id="ItemReader" class="com.syc.studio.reader.StudioReader" scope="step">
<property name="context" ref="BatchContext" />
<property name="areaDao" ref="AreaDao" />
</bean>
And here is the job definition in xml:
<bean id="StudioJob" class="org.springframework.batch.core.job.SimpleJob">
<property name="steps">
<list>
<bean id="StudioStep" parent="SimpleStep" >
<property name="itemReader" ref="ItemReader"/>
<property name="itemWriter" ref="ItemWriter"/>
<property name="retryableExceptionClasses">
<map>
<entry key="com.syc.studio.exception.CustomException" value="true"/>
</map>
</property>
<property name="retryLimit" value="2" />
</bean>
</list>
</property>
<property name="jobRepository" ref="jobRepository" />
</bean>
Author:
public void write(List<? extends Object> obj) throws Exception {
List<Studio> list = (List<Studio>) obj.get(0);
for (int i = 0; i <= list.size(); i++) {
Studio studio = list.get(i);
if (apiClient == null) {
apiClient = new APIClient("v2");
}
this.uploadXML(studio);
}
Read method after the sentence from @ holi-java:
public List<Studio> read() throws Exception {
if (this.listIterator == null) {
this.listIterator = initializing();
}
return this.listIterator.hasNext() ? this.listIterator.next() : null;
}
private Iterator<List<Studio>> initializing() {
List<Studio> listOfStudiosFromApi = hopsService.getStudioLocations();
for (Studio studio : listOfStudiosFromApi) {
log.info("Studio being read: {}", studio.getCode());
List areaList = areaDao.getArea(studio.getCode());
if (areaList.size() > 0) {
studio.setArea((String) areaList.get(0));
log.info("Area {1} is fetched for studio {2}", areaList.get(0), studio.getCode());
}
this.getContext().setReadCount(1);
}
return Collections.singletonList(listOfStudiosFromApi).iterator();
}
source to share
spring - package documentation for ItemReader.read assert:
Implementations must return null at the end of the input dataset.
But your read method always returns a list and should look like this:
public Studio read() throws Exception {
if (this.results == null) {
List<Studio> list = hopsService.getStudioHops();
...
this.results=list.iterator();
}
return this.results.hasNext() ? this.results.next() : null;
}
if you want your read method to return a list, you have to push the results like this:
public List<Studio> read() throws Exception {
List<Studio> results=hopsService.getStudioHops(this.page++);
...
return results.isEmpty()?null:results;
}
if you are unable to display results from the Service, you can solve it as follows:
public List<Studio> read() throws Exception {
if(this.results==null){
this.results = Collections.singletonList(hopsService.getStudioHops()).iterator();
}
return this.results.hasNext()?this.results.next():null;
}
it is better not to read the list of elements List<Studio>
, read the element instead Studio
. when you read the list of elements you may have duplicated the logic iteration between writers
and processors
as you showed the demo in the comments. if you have a huge list of data to process, you can combine pagination in your reader like:
public Studio read() throws Exception {
if (this.results == null || !this.results.hasNext()) {
List<Studio> list = hopsService.getStudioHops(this.page++);
...
this.results=list.iterator();
}
return this.results.hasNext() ? this.results.next() : null;
}
Perhaps you need to see the mechanism for handling steps .
- ItemReader - read an item at a time.
- ItemProcessor - processing an item at the same time.
- ItemWriter - Write the entire fragment of out items .
source to share