Shulou(Shulou.com)05/31 Report--

This article focuses on "how to process data from Web services to MongoDB". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to process data from Web services to MongoDB.

An overview

How to create a Spring Batch application that uses Web service data and inserts it into a MongoDB database.

Request

Developers reading this article must be familiar with Spring Batch (examples) and MongoDB.

Environment

The Mongo database is deployed in MLab. Please follow the steps in this Quick start.

Batch applications are deployed in Heroku PaaS. Please see here for details.

IDE STS or IntelliJ or Eclipse.

Java 8 JDK .

Note: batches can also be run locally.

Script

The global scene steps are:

Read data from the Web service, in this case: https://sunrise-sunset.org/api

Get the coordinates of the list of cities, and then call API to read the sunrise and sunset dates and times.

two。 Process data and extract business data

Business processing of data collection

3. Insert processed data into MongoDB

Save the processed data as an mongo document

Coding

Input: a list of city data in JSON format in the local file, as follows:

[{"name": "Danemark", "City": [{"name": "Copenhague", "lat": 55.676098, "lng": 12.568337, "timeZone": "CET"}, {"name": "Aarhus", "lat": 56.162939 "lng": 10.203921, "timeZone": "CET"}, {"name": "Odense", "lat": 55.39594, "lng": 10.38831, "timeZone": "CET"}, {"name": Aalborg "lat": 57.046707, "lng": 9.935932, "timeZone": "CET"}]

Our scenario gets the input data from the local json file. The mapping bean is as follows:

Guodou:

Import java. Io . Serializable; import java. Util . List

Import com. Fastxml . Jackson. Comments. JsonIgnoreProperties; @ JsonIgnoreProperties (ignoreUnknown = true) public class BCountry implements Serializable {private static final long serialVersionUID = 1L; private string name; private list

< BCity >

City; public BCountry () {super ();} public BCountry (String name,List

< BCity >

Cities) {super (); this. Name = name; this. City = city;} public BCountry (String name) {super (); this. Name = name;} public String getName () {return name;} public void setName (String name) {this. Name = name;} public List

< BCity >

GetCities () {return to the city;} public void setCities (List

< BCity >

Cities) {this. City = city;} @ covers public int hashCode () {final int prime = 31; int result = 1; result = gold * result + ((city = = empty)? 0: city. HashCode (); result = gold * result + ((name = = empty)? 0: name. HashCode (); returns the result;} @ overrides public boolean equals (Object obj) {if (this = = obj) returns true; if (obj = = null) returns false; if (getClass ()! = OBJ. GetClass () returns false; BCountry other = (BCountry) obj; if (cities = = null) {if (other). The city! = null) returns false;} otherwise if (! City. Equality, etc. City) returns false; if (name = = null) {if (other). Name! = null) returns false;} otherwise if (! First name. Equality (others. Return false; return true;} @ overwrite public String toString () {return "BCountry [name =" + name + ", cities =" + cities + "]";}}

And City beans:

Import java. Io . Serializable; imported com. Fastxml . Jackson. Comments. JsonIgnoreProperties; @ JsonIgnoreProperties (ignoreUnknown = true) Public class BCity implements Serializable {private static final long serialVersionUID = 1L; private String name,timeZone; Private double Lat, lng; public BCity () {super ();} public BCity (String name,String timeZone,double lat,double lng) {super (); this. Name = name; this. TimeZone = timeZone; this. Lat = lat; this. Lng = lng;} public String getName () {return name;} public void setName (String name) {this. Name = name;} public String getTimeZone () {return time zone;} public void setTimeZone (String timeZone) {this. TimeZone = timeZone;} public double getLat () {returns latitude;} public void setLat (double lat) {this. Lat = lat;} public double getLng () {returns lng;} public void setLng (double lng) {this. Lng = lng;} @ overrides public String toString () {returns "BCity [name =" + name + ", timeZone =" + timeZone + ", lat =" + lat + ", lng =" + lng + "]";} @ overwrites public int hashCode () {final int prime = 31; int result = 1; long temperature; temp = Double. DoubleToLongBits (lat); result = prime * result + (int) (temp ^ (temp > > 32)); temp = Double. DoubleToLongBits (lng); result = prime * result + (int) (temp ^ (temp > 32)); result = gold * result + ((name = = empty)? 0: name. HashCode (); result = prime * result + ((timeZone = = empty)? 0: timeZone. HashCode (); returns the result;} @ overrides public boolean equals (Object obj) {if (this = = obj) returns true; if (obj = = null) returns false; if (getClass ()! = OBJ. GetClass () returns false; BCity other = (BCity) obj; if (double). DoubleToLongBits (LAT)! = double. DoubleToLongBits (others). LAT) returns false; if (double). DoubleToLongBits (LNG)! = double. DoubleToLongBits (others). LNG) returns false; if (name = = null) {if (other). Name! = null) returns false;} otherwise if (! First name. Equality (others. Return false; if (timeZone = = null) {if (other). TimeZone! = null) returns false;} otherwise if (! TimeZone. Equality (others. TimeZone) returns false; returns true;}}

The batch reader implements @ LineMapper. You can adapt the reader to our data source (example):

Import java. Util . List; import organization. Spring frame. Batches. Project. archives. LineMapper; imported com. Ahajri . Batches. Beans. BCountry; imported com. Fastxml . Jackson. Data binding. ObjectMapper; imported com. Fastxml . Jackson. Data binding. Type. CollectionType; the public class BCountryJsonLineMapper implements LineMapper

< List < BCountry >

> {private final ObjectMapper mapper = new ObjectMapper (); @ overwrite public List

< BCountry >

MapLine (String line,int lineNumber) throws Exception {CollectionType collectionType = mapper. GetTypeFactory (). ConstructCollectionType (list. Class, BCountry. Class); returns the mapper. ReadValue (line,collectionType);}}

When processing data batches, check whether the business processing data of a city on the same day has been saved in the database. The way to search for data in MongoDB is in this detailed post.

ItemProcessor converts @ BCountry objects to MongoDB Document objects. The process is described in detail as follows:

Implementing ItemProcessor with public class BCountryPrayTimeEventItemProcessor

< List < BCountry >

, List

< Document >

> {private static final String EVENTS_COLLECTION_NAME = "event"; private static final Logger LOG = LoggerFactory. GetLogger (BCountryPrayTimeEventItemProcessor. Class); @ Autowiredprivate PrayTimeService prayTimeService; @ Autowiredprivate CloudMongoService cloudMongoService; @ override public List

< Document >

Process (List

< BCountry >

Items) throws Exception {final List

< Document >

Docs = new ArrayList (); item. Stream (). ForEach (item-> {final String countryName = item. GetName (); project. GetCities (). Stream (). ForEach (c-> {final Document prayTimeCityEventDoc = new Document (); / / cycle the city and extract the prayer time final String cityName = c for today. GetName (); final String cityTimeZone = c. GetTimeZone (); final double lat = c. GetLat (); final double lng = c. GetLng (); final LocalDateTime nowOfCity = LocalDateTime. Now (zoneid). (cityTimeZone); final QueryParam [] queryParams = new QueryParam [5]; queryParams [0] = new QueryParam ("CITY_NAME", OperatorEnum. EQ . Name (), cityName of); queryParams [1] = new QueryParam ("EVENT_TYPE", OperatorEnum. EQ . Name (), event type. PRAY_TIME . Name (); queryParams [2] = new QueryParam ("month", OperatorEnum. EQ . Name (), nowOfCity. GetMonthValue (); queryParams [3] = new QueryParam ("DAY_OF_MONTH", OperatorEnum. EQ . Name (), nowOfCity. GetDayOfMonth (); queryParams [4] = new QueryParam ("COUNTRY_NAME", OperatorEnum. EQ . Name (), country name); List

< Document >

FoundEvents = null; try {foundEvents = cloudMongoService. Search for (EVENTS_COLLECTION_NAME,queryParams);} catch (BusinessException E1) {records. Error ("City Prayer time not found" + cityName + "right" + nowOfCity. GetDayOfMonth () + "/" + nowOfCity. GetMonthValue ();} try {if (CollectionUtils). IsEmpty (foundEvents)) {/ / Prayer time has not yet created a prayTimeCityEventDoc. Put ("country_name", countryName); prayTimeCityEventDoc. Put ("city_name", cityName); prayTimeCityEventDoc. Put ("EVENT_TYPE", event type. PRAY_TIME . Name (); prayTimeCityEventDoc. Put ("relapse", RecurringEnum. YEARLY . Name (); prayTimeCityEventDoc. Put ("month", nowOfCity. GetMonthValue (); prayTimeCityEventDoc. Put ("DAY_OF_MONTH", nowOfCity. GetDayOfMonth (); prayTimeCityEventDoc. Put ("lat", lat); prayTimeCityEventDoc. Put ("lng", lng); prayTimeCityEventDoc. Put ("CREATION_DATE", HCDateUtils. ConvertToDateViaSqlTimestamp (nowOfCity)); final Map

< String，Object >

PrayInfos = prayTimeService. GetPrayTimeByLatLngDate (lat,lng, date. From (nowOfCity. AtZone (zoneid. (cityTimeZone). ToInstant (), cityTimeZone); prayTimeCityEventDoc. Put ("pray_infos", file. Parse (toJSON (prayInfos) of the new GSON ()); docs. Add (prayTimeCityEventDoc);} else {record. Information (string. Format ("= = > the time of prayer already exists in cities:% S, month:% d, day:% d", cityName,nowOfCity. GetMonthValue (), nowOfCity. GetDayOfMonth ());}} catch (BusinessException e) {record. Error ("there is a problem calculating prayer time:", e); throw a new RuntimeException (e);}}); return document;}}

Batch configuration class:

@ configuration @ EnableBatchProcessing@EnableScheduling public class BatchConfiguration {private static final String SCANDINAVIAN_COUNTRIES_JSON_FILE = "scandinavian-countries.json"; private static final String EVENT_COLLECTION_NAME = "event_collection"; private static final Logger LOG = LoggerFactory. GetLogger (BatchConfiguration. Class); @ Autowiredprivate JobBuilderFactory jobBuilderFactory; @ Autowiredprivate StepBuilderFactory stepBuilderFactory; @ Autowired private MLabMongoService mlabMongoService; @ bean public ResourcelessTransactionManager transactionManager () {returns the new ResourcelessTransactionManager ();} @ bean public MapJobRepositoryFactoryBean mapJobRepositoryFactory (ResourcelessTransactionManager txManager) throws the exception {MapJobRepositoryFactoryBean factory = new MapJobRepositoryFactoryBean (txManager); factory. AfterPropertiesSet (); returns to the factory;} @ bean public JobRepository jobRepository (MapJobRepositoryFactoryBean factory) throws an exception {return (JobRepository) factory. GetObject ();} private SimpleJobLauncher jobLauncher

@ public SimpleJobLauncher jobLauncher (JobRepository jobRepository) {jobLauncher. SetJobRepository (jobRepository); return jobLauncher;} @ PostConstructprivate void initJobLauncher () {jobLauncher = new SimpleJobLauncher ();} @ bean FlatFileItemReader

< List < BCountry >

> reader () {FlatFileItemReader

< List < BCountry >

> reader = new FlatFileItemReader (); reader. SetName ("scandinaviandCountriesReader"); reader. SetResource (new ClassPathResource (SCANDINAVIAN_COUNTRIES_JSON_FILE)); readers. SetLineMapper (new BCountryJsonLineMapper ()); report back to the reader;} @ public ItemWriter

< List < Document >

> writer () {return the new ItemWriter

< List < Document >

> () {@ overwrite public void write (List > items) throw Exception {try {if (! CollectionUtils . IsEmpty (project) & & project. Size () > 0) {List

< Document >

FlatDocs = items. Stream (). FlatMap (List:: stream). Collect (collector). ToList (); mlabMongoService. InsertMany (EVENT_COLLECTION_NAME,flatDocs);} else {record. Warning ("No event can be saved.");}} catch (BusinessException e) {throw a new RuntimeException (e);};}

@ bean public BCountryTimeEventItemProcessor processor () {return the new BCountryTimeEventItemProcessor ();}

@ bean public Job scandvTimeJob () {returns jobBuilderFactory. Get ("scandvTimeJob"). Incrementmenter (new RunIdIncrementer ()). Process (step1 ()). End (). Build ();}

@ bean public Step step1 () {returns stepBuilderFactory. Get ("step1").

< List < BCountry >

, List

< Document >

Chunk (10). Reader (reader ()). Processor (processor ()). Writer (writer). Build ();} / / end:: jobstep []

/ / 15 minutes at midnight every day @ Scheduled (cron = "0150 *") public void startScandvEventTimeJob () throws Exception {record. Info ("= = > work start time:" + new date ()); JobParameters param = new JobParametersBuilder (). AddString ("Job ID", string. ValueOf (system). CurrentTimeMillis ()). ToJobParameters (); JobExecution execution = jobLauncher. Run (scandvPrayTimeJob (), param); record. Information ("= = > work completed status:" + execution. GetStatus ();}} of

Deploy de Batch to Heroku:

Git add . Git commit-m "Deploy Batch" git push heroku master

Note: to disable the default batch startup, add this to application.yml

S p r i ng: b a t c h: j o b: subscribe a b-liter é d: F-liter trumpet ë here, I believe you have a better understanding of "how to process data from Web services to MongoDB". You might as well do it in practice! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.