Category Archives: Uncategorized

Functions as Arguments Java vs Scala, Game Set Match Scala Wins!

Uncategorized,

This is how you would create a function that takes a function as argument in Java

import java.util.function.Function;

class Scratch {

    public static void doCallFunc(int num, Function<Integer,String> fn) {
        System.out.println( "Result : "+fn.apply( num ) );
    }

    public static void main(String[] args) {
        Function<Integer,String> myFunc = num -> "Value = " + num;
        System.out.println( myFunc.apply( 7 ) );        
    }
}

The Function<A,B> myFunc = num -> “Value = ” + num;
Here :
A = the type of the first argument, in this example an interger
B = the type of the result/returned value, in this exampel a String

And for multiple parameters you need to create an interface like this

import java.util.function.Function;


@FunctionalInterface
interface TwoParamFunction<A,B,C> {
    public C apply(A a, B b);
}

class Scratch {

    public static void doCallFunc2(int num, TwoParamFunction<Integer,String,String> fn) {
        System.out.println( "Result : "+fn.apply( num, "Value" ) );
    }

    public static void main(String[] args) {
        TwoParamFunction<Integer,String,String> myFunc2 = (num,str) -> str + " : " + num;
        doCallFunc2( 7, myFunc2 );

    }
}

Now with TWO (2) parameters instead, it looks looks alot more complicated.
TwoParamFunction<A,B,C>
A = is the type of the first parameter
B = is the type of the second parameter
C = is the type of the resule/returned value

If we look at Scala the code looks alot simpler and much more intuitive

def myFunc( num:Int ):String = {
 "Value = " + num
}

def doCallFunc( num:Int, fn:(Int)=>String ):Unit = {
 println("Result :"+fn(num))
}

doCallFunc(123,myFunc)

here the definition of the function
fn:(Int)=>String
clearly spells out that the first argument is an Int and the return type is a String.

And if we in Scala would have 2 or more arguments you have probably already guessed it

def myFunc2( num:Int, str:String ):String = {
  str + num
}

def doCallFunc2( num:Int, fn:(Int,String)=>String ):Unit = {
  println("Result :"+fn(num,"Value = "))
}

doCallFunc2( 123, myFunc2 )

For functions as arguments/parameters example above, then Scala wins all week !

Over and out !

Apache Zeppelin, with Spark and Cassandra, the perfect tool

Uncategorized

Zeppelin has become one of my favourite tools in my toolbox. I am heavily designing stuff for Cassandra and in Scala, and even though I love Cassandra there are times when things just gets so complicated with the CQL command line, and creating a small project in IntelliJ just seems like too much hazel. Then using Zeppelin to try out is just perfect. So this page is a How-To with some useful Cookbook recipes.

Setting Up Zeppelin

I use Docker where things are so much easier, and I pick v0.8.0 cause I never got 0.8.2 to work for some reason.

Download and Start Cassandra

 

Download and Start Zeppelin

Download Zeppelin image

Start Zeppelin on port 8080

-p hp:cp
hp = Host Port, the port on your local machine
cp = Container Port, the port inside the docker which is what Zeppelin is exposing

Go to localhost:8080 in your web browser and you should see something like this

Setup Zeppelin

Find out the IP address of Cassandra in you Docker network, as you can see of the inspect, the IP address is 172.17.0.3.

 

Set up IP address for Cassandra in the Spark Interpreter

Go to the section on “Spark”

Now add a row that says

Now also edit the Dependencies

You can do this in many ways, either you specify the MAVEN repo with version OR you download the JAR file(s) to disk and copy them into the Docker. I had to do the latter due to some issue with my network.

You need these two libraries :

Simply click on the JAR file and download the file, then copy it into the docker with

Setup IP address for Cassandra in Cassandra Interpreter

Create your first Notebook

Cookbook Recipes

Load Table into RDD and count rows

This is just to show how you load a table into an RDD, once in the RDD you can play around with it and do lots of stuff.

Show key spaces using the built in Cassandra interpreter using CQL

The result :

Create Keyspace and Table using CQL

Insert data by hand using CQL

Fill the table with bogus data using Spark and Scala

 

Select data using CQL

Create VIEW so that we can run SQL

 

Run SQL, ohh sweet SQL 🙂

By creating temporary views like this, we can also do joins if we would like to.

Obviously this is not how Cassandra was intended to be used, but the point here is more of giving the ability to troubleshoot, turist around in the data with ease instead of setting up a project, and do the joins inside of the code. Here we are able to really trail and error until we get what we want.

That was all for now

-Tobias

SQL LIKE operation in Cassandra, is possible in v3.4+

Uncategorized

For a long time it has not been possible to do a SELECT * FROM table WHERE firstname like ‘t%’; in Cassandra like you could in eg.. MySQL or any other Relation Database for that matter.

In Cassandra v3.4 this is now possible, BUT it requires some extra to do it right, and that is why I created this blog post cause I had trouble finding it.

The solution is to create a separate index, and not the secondary indexes that Cassandra came with, but a different index, called a SASI index.

This is what I have

And the content of it looks like this

And now I would like to search for all the rows that has a first name that starts with a ‘t’

In SQL that would have been :

SELECT * FROM bth.employee WHERE firstname LIKE ‘t%’;

In fact we could have done that on any column …. but in Cassandra it would result in something like this:

In Cassandra we first has to decide on which columns this should be possible, by creating an index like this:

And so you can now do the following

But what if you decide that I would like to know all the employees that ends with an ‘s’ in their name, so something like this:

So to be able to search for something that contains we have to change the index like this instead:

And now you can run that query again:

You can read more about the SASI index here https://docs.datastax.com/en/cql/3.3/cql/cql_reference/refCreateSASIIndex.html

Enjoy!

-Tobias

Apache SPARK and Cassandra and SQL

Uncategorized, , , , , , , , ,

This is a short intro to start using Apache SPARK with Cassandra, running SQL on the Cassandra tables.

Note that I am not running a SPARK cluster, I am running “local”, to me this is really convenient, not having to run a SPARK server and workers for something so small. So for playing around with SPARK and Cassandra this is really good.

I am using Scala and SBT.

Something I was struggling hard with, to get the dependency versions right. It is crucial that you do not do like I did first, use version 1.5.2 of Spark and 1.5.0 for SparkCassandraConnector, this will NOT work. I constantly got exception with java.lang.NoSuchMethodException, so incredibly frustrating to try out version after version.

build.sbt

A small Scala program to show how it works

SparkTest.scala

The output…

 

 

Create an MBean (JMX) in Scala

Uncategorized

Create the MBean like this

NOTE, that the interface/trait must end with MBean in the name

And this is how you register your MBean

And the simply launch Java Mission Control (imc), attach to the JVM, and modify your MBean attributes as you like.

Enjoy!

SBT module not found, why ?

Uncategorized

I have an build.sbt file that looks like this

But for some reason I can’t get slf4j downloaded from the Maven repository (http://mvnrepository.com)

If I search the Maven Repository, I can clearly see that the version I intend to use is there.

Running “sbt compile” from the command line will result in the following output, and here it is time to pay attention to the details, look at what it is trying to do !!!

As you can see above the package (jar) it tries to download is not slf4j-api it is slf4j-api_2.11;1.7.10

The build.sbt file uses the double and single % (percent) character and this is what makes the difference.  The %% makes SBT append the project specified scala version to the package name, resulting in a name “slf4j-api_2.11”.

BUT that name does not exist in the Maven Repository, however, the “slf4j-api” does

So by simply choosing one instead of two %, the problem will go away 🙂

Thus, the built.sbt should look like this instead

For reference go to https://www.playframework.com/documentation/2.1.1/SBTDependencies to read more about how SBT handles decencies and the % and %%.

I read this article to finally get this right http://stackoverflow.com/questions/17461453/build-scala-and-symbols-meaning

See who is connected to the same network / WIFI

Uncategorized

There are tools out there, such as Angry IP Scanner, iNet or IP Scanner,  but this can easily be done using a bash script, not as nice looking perhaps but it does the job.

The script :

Which should give you an output that looks like this

That’s it

-Tobias