Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to make PigPen support native code and namespace". The explanation in this article is simple and clear and easy to learn and understand. Please follow the editor's train of thought to study and learn "how to make PigPen support native code and namespace".

In the article introducing PigPen, the author mentioned in the Future Work section that we cannot call locally declared code in PigPen. For example, the following code cannot be executed in the final generated Pig script and will report an error that the symbol foo cannot be found:

(ns test.core (: require [pigpen.core: as pig])) (defn foo [x] …) (pig/map foo)

In this way, you can only write all the things that the foo function does after pig/map. If foo has a lot of work to do and the code is long, the code written will be very ugly.

PigPen not only does not support native code calls, but also does not support references to namespace (https://groups.google.com/forum/#!msg/pigpen-support/-Kd06UfzxEU/vYEAZvmZLFcJ).

These features, which seem natural in any programming language, are not supported in the initial PigPen. In fact, every native code call here or other namespace function calls are finally written into a Pig script is a UDF, and one of the purposes of PigPen is to replace the Pig script and UDF mixed way, so there should be a solution, but there is no time (and lazy) to pick the source code to think of a way to deal with this situation, so go to GitHub to find the main contributor Matt Bossenbroek, send an email to him for advice.

Matt points out that if you want to call native code, you need to do this:

(ns test.core (: require [pigpen.core: as pig])) (defn foo [x] …) (pig/map (do (require 'test.core) foo))

I tried it and didn't seem to be able to achieve the desired results.

So I tried to change the generated Pig script to see if it would work. Anyway, the generated Pig script file is also used when the final deployment is run on the cluster. In the generated Pig script, all the Clojure code is wrapped by a set of pigpen.PigPenFn* classes (PigPenFnBoolean,PigPenFnString,PigPenFnTuple,PigPenFnDataBag,PigPenFnDataByteArray) as UDF and inserted into the resulting Pig script, and different suffixes return values of different Pig base types.

In the generated Pig script, the first parameter of these classes is'(clojure.core/require (quote [pigpen.pig]))'. If you look at the code, you can see that this parameter will be read in as Clojure code for initialization, so you should add the namespace to be included here. I tried some and sure enough it worked. Take another look at the code and find that it is not easy to add this require through the PigPen method, so you plan to generate a Pig script with PigPen first, and then add another namespace to be included here (it won't be too troublesome to write a script to add namespace to each of these classes).

Also by the way, ask Matt how to use PigPen to do this, and Matt said that he is considering how to support referencing namespace. As a result, on the morning of the second day of the Lunar New year, I received an email saying that the new version of PigPen ([com.netflix.pigpen/pigpen "0.1.4]) could be supported, and an example was given:

(ns pigpen-demo.core (: require [pigpen.core: as pig] [clojure.string: as str])) (defn square [x] (* x)) (defn my-query [] (- > > (pig/return [1 23]) (pig/map square) (pig/map (fn [x] (square x) (pig/into []) (pig/map # (str/join ") "%) (pig/dump)

But be sure to package the entire project into a uberjar and distribute it to the cluster. This basically allows me to use PigPen in my experimental work (the production environment does not allow the misuse of new things).

Thank you for your reading, the above is the content of "how to make PigPen support native code and namespace". After the study of this article, I believe you have a deeper understanding of how to make PigPen support native code and namespace. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.