I'm developing a Spark Application and I'm used to Spring as a Dependency Injection Framework. Now I'm stuck with the problem, that the processing part uses the @Autowired functionality of Spring, but it is serialized and deserialized by Spark.
So the following code gets me into trouble:
Processor processor = ...; // This is a Spring constructed object
// and makes all the trouble
JavaRDD<Txn> rdd = ...; // some data for Spark
rdd.foreachPartition(processor);
The Processor looks like that:
public class Processor implements VoidFunction<Iterator<Txn>>, Serializeable {
private static final long serialVersionUID = 1L;
@Autowired // This will not work if the object is deserialized
private transient DatabaseConnection db;
@Override
public void call(Iterator<Txn> txns) {
... // do some fance stuff
db.store(txns);
}
}
So my question is: Is it even possible to use something like Spring in combination with Spark? If not, what is the most elegant way to do something like that? Any help is appreciated!
Aucun commentaire:
Enregistrer un commentaire