Remove the cardo-updater agent from OSX

Posted on June 5, 2018 by tobias

I have the intercom from Cardo Systems, and it is really good
BUT when I updated the firmware some time agoe, it decided to install some software that takes port 8080, which is one of those really common ports used by a lot of applications out there. So it really becomes a problem…

Now I figured this out after using lsof

Tobiass-MacBook-Pro-2:~ tobias$ sudo lsof -i tcp:8080
Password:
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
cardo-upd 107 root    6u  IPv4 0xa6a0fdd48cbb4495      0t0  TCP localhost:http-alt (LISTEN)

Tobiass-MacBook-Pro-2:~ tobias$ sudo lsof -i tcp:8080

Password:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

cardo-upd 107 root 6u IPv4 0xa6a0fdd48cbb4495 0t0 TCP localhost:http-alt (LISTEN)

Then I got the PID, so now I could do a ps -ef, to figure out WHICH parent process started it.

Tobiass-MacBook-Pro-2:~ tobias$ ps -ef | grep 107
    0   107     1   0 Sat09PM ??        16:56.45 /usr/local/cardo-updater

1 2	Tobiass-MacBook-Pro-2:~ tobias$ ps -ef \| grep 107 0 107 1 0 Sat09PM ?? 16:56.45 /usr/local/cardo-updater

Ohh PPID = 1 🙂 That is the launchd-process

Tobiass-MacBook-Pro-2:~ tobias$ ps -ef | head
  UID   PID  PPID   C STIME   TTY           TIME CMD
    0     1     0   0 Sat09PM ??         5:09.23 /sbin/launchd

Tobiass-MacBook-Pro-2:~ tobias$ ps -ef | head

UID PID PPID C STIME TTY TIME CMD

0 1 0 0 Sat09PM ?? 5:09.23 /sbin/launchd

OK so now we know it is the launchd process.
So first just find it in launchd

Tobiass-MacBook-Pro-2:~ tobias$ sudo launchctl list | grep cardo
20177 0 cardo-updater

1 2	Tobiass-MacBook-Pro-2:~ tobias$ sudo launchctl list \| grep cardo 20177 0 cardo-updater

Now in order to unload it we need to find the path to the plist file

Tobiass-MacBook-Pro-2:~ tobias$ sudo launchctl procinfo 20724
program path = /usr/local/cardo-updater
mach info = {
	task-kernel port = 0xb03 (unknown)
	task-host port = 0xc03 (host)
	task-name port = 0xf03 (unknown)
	task-bootstrap port = 0xf07 (unknown)
	task-seatbelt port = 0x0 (unknown)
	task-access port = 0xf0b (unknown)
	task-debug control port = 0xf0f (unknown)
}
argument count = 1
argument vector = {
	[0] = /usr/local/cardo-updater
}
environment vector = {
	PATH => /usr/bin:/bin:/usr/sbin:/sbin
	XPC_SERVICE_NAME => cardo-updater
	XPC_FLAGS => 0x1
}
bsd proc info = {
	pid = 20724
	unique pid = 20724
	ppid = 1
	pgid = 20724
	status = stopped
	flags = session leader
	uid = 0
	svuid = 0
	ruid = 0
	gid = 0
	svgid = 0
	ruid = 0
	comm name = cardo-updater
	long name = cardo-updater
	controlling tty devnode = 0xffffffff
	controlling tty pgid = 0
}
audit info
	session id = 100000
	uid = 4294967295
	success mask = 0x0
	failure mask = 0x0
	flags = is_initial
sandboxed = no
container = (no container)

responsible pid = 20724
responsible unique pid = 20724
responsible path = /usr/local/cardo-updater

pressured exit info = {
	dirty state tracked = 0
	dirty = 0
	pressured-exit capable = 0
}

jetsam priority = 3: background
jetsam memory limit = -1
jetsam state = (normal memory state)

entitlements = (no entitlements)

code signing info = valid
	platform dyld

cardo-updater = {
	active count = 1
	path = /Library/LaunchDaemons/com.cardosystems.cardo-updater.plist
	state = running

	program = /usr/local/cardo-updater
	arguments = {
		/usr/local/cardo-updater
	}

	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		XPC_SERVICE_NAME => cardo-updater
	}

	domain = com.apple.xpc.launchd.domain.system
	minimum runtime = 10
	exit timeout = 5
	runs = 4
	successive crashes = 0
	excessive crashing = 0
	pid = 20724
	immediate reason = inefficient
	forks = 0
	execs = 1
	trampolined = 1
	started suspended = 0
	proxy started suspended = 0
	last exit code = 0

	event triggers = {
	}

	endpoints = {
	}

	dynamic endpoints = {
	}

	pid-local endpoints = {
	}

	instance-specific endpoints = {
	}

	event channels = {
	}

	sockets = {
	}

	spawn type = daemon
	spawn role = (null)
	jetsam priority = 3
	jetsam memory limit (active) = (unlimited)
	jetsam memory limit (inactive) = (unlimited)
	jetsamproperties category = daemon
	cpumon = default

	properties = {
		partial import = 1
		launchd bundle = 0
		xpc bundle = 0
		keepalive = 1
		runatload = 0
		dirty at shutdown = 0
		low priority i/o = 0
		low priority background i/o = 0
		legacy timer behavior = 0
		exception handler = 0
		multiple instances = 0
		supports transactions = 0
		supports pressured exit = 0
		enter kdp before kill = 0
		wait for debugger = 0
		app = 0
		system app = 0
		creates session = 0
		inetd-compatible = 0
		inetd listener = 0
		abandon process group = 0
		one-shot = 0
		requires reap = 0
		event monitor = 0
		penalty box = 0
		pended non-demand spawn = 0
		role account = 0
		launch only once = 0
		system support = 0
		app-like = 0
		inferred program = 1
		joins gui session = 0
		joins host session = 0
		parameterized sandbox = 0
		resolve program = 0
		abandon coalition = 0
		extension = 0
		nano allocator = 0
		no initgroups = 0
		start on fs mount = 0
		endpoints initialized = 1
		disallow all lookups = 0
		system service = 0
	}
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

Tobiass-MacBook-Pro-2:~ tobias$ sudo launchctl procinfo 20724

program path = /usr/local/cardo-updater

mach info = {

task-kernel port = 0xb03 (unknown)

task-host port = 0xc03 (host)

task-name port = 0xf03 (unknown)

task-bootstrap port = 0xf07 (unknown)

task-seatbelt port = 0x0 (unknown)

task-access port = 0xf0b (unknown)

task-debug control port = 0xf0f (unknown)

}

argument count = 1

argument vector = {

[0] = /usr/local/cardo-updater

}

environment vector = {

PATH => /usr/bin:/bin:/usr/sbin:/sbin

XPC_SERVICE_NAME => cardo-updater

XPC_FLAGS => 0x1

}

bsd proc info = {

pid = 20724

unique pid = 20724

ppid = 1

pgid = 20724

status = stopped

flags = session leader

uid = 0

svuid = 0

ruid = 0

gid = 0

svgid = 0

ruid = 0

comm name = cardo-updater

long name = cardo-updater

controlling tty devnode = 0xffffffff

controlling tty pgid = 0

}

audit info

session id = 100000

uid = 4294967295

success mask = 0x0

failure mask = 0x0

flags = is_initial

sandboxed = no

container = (no container)

responsible pid = 20724

responsible unique pid = 20724

responsible path = /usr/local/cardo-updater

pressured exit info = {

dirty state tracked = 0

dirty = 0

pressured-exit capable = 0

}

jetsam priority = 3: background

jetsam memory limit = -1

jetsam state = (normal memory state)

entitlements = (no entitlements)

code signing info = valid

platform dyld

cardo-updater = {

active count = 1

path = /Library/LaunchDaemons/com.cardosystems.cardo-updater.plist

state = running

program = /usr/local/cardo-updater

arguments = {

/usr/local/cardo-updater

}

default environment = {

PATH => /usr/bin:/bin:/usr/sbin:/sbin

}

environment = {

XPC_SERVICE_NAME => cardo-updater

}

domain = com.apple.xpc.launchd.domain.system

minimum runtime = 10

exit timeout = 5

runs = 4

successive crashes = 0

excessive crashing = 0

pid = 20724

immediate reason = inefficient

forks = 0

execs = 1

trampolined = 1

started suspended = 0

proxy started suspended = 0

last exit code = 0

event triggers = {

}

endpoints = {

}

dynamic endpoints = {

}

pid-local endpoints = {

}

instance-specific endpoints = {

}

event channels = {

}

sockets = {

}

spawn type = daemon

spawn role = (null)

jetsam priority = 3

jetsam memory limit (active) = (unlimited)

jetsam memory limit (inactive) = (unlimited)

jetsamproperties category = daemon

cpumon = default

properties = {

partial import = 1

launchd bundle = 0

xpc bundle = 0

keepalive = 1

runatload = 0

dirty at shutdown = 0

low priority i/o = 0

low priority background i/o = 0

legacy timer behavior = 0

exception handler = 0

multiple instances = 0

supports transactions = 0

supports pressured exit = 0

enter kdp before kill = 0

wait for debugger = 0

app = 0

system app = 0

creates session = 0

inetd-compatible = 0

inetd listener = 0

abandon process group = 0

one-shot = 0

requires reap = 0

event monitor = 0

penalty box = 0

pended non-demand spawn = 0

role account = 0

launch only once = 0

system support = 0

app-like = 0

inferred program = 1

joins gui session = 0

joins host session = 0

parameterized sandbox = 0

resolve program = 0

abandon coalition = 0

extension = 0

nano allocator = 0

no initgroups = 0

start on fs mount = 0

endpoints initialized = 1

disallow all lookups = 0

system service = 0

}

As you can see above the plist file is :

path = /Library/LaunchDaemons/com.cardosystems.cardo-updater.plist

Allright, so now we can unload it

Tobiass-MacBook-Pro-2:~ tobias$ sudo launchctl unload /Library/LaunchDaemons/com.cardosystems.cardo-updater.plist

1	Tobiass-MacBook-Pro-2:~ tobias$ sudo launchctl unload /Library/LaunchDaemons/com.cardosystems.cardo-updater.plist

And that is it !

-Tobias

SQL LIKE operation in Cassandra, is possible in v3.4+

Posted on August 12, 2016 by tobias

For a long time it has not been possible to do a SELECT * FROM table WHERE firstname like ‘t%’; in Cassandra like you could in eg.. MySQL or any other Relation Database for that matter.

In Cassandra v3.4 this is now possible, BUT it requires some extra to do it right, and that is why I created this blog post cause I had trouble finding it.

The solution is to create a separate index, and not the secondary indexes that Cassandra came with, but a different index, called a SASI index.

This is what I have

CREATE TABLE bth.employee (
    id int,
    lastname text,
    firstname text,
    dateofbirth date,
    PRIMARY KEY (id, lastname, firstname)
) WITH CLUSTERING ORDER BY (lastname ASC, firstname ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'bucket_high': '1.5', 'bucket_low': '0.5', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'enabled': 'true', 'max_threshold': '32', 'min_sstable_size': '50', 'min_threshold': '4', 'tombstone_compaction_interval': '86400', 'tombstone_threshold': '0.2', 'unchecked_tombstone_compaction': 'false'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE bth.employee (

id int,

lastname text,

firstname text,

dateofbirth date,

PRIMARY KEY (id, lastname, firstname)

) WITH CLUSTERING ORDER BY (lastname ASC, firstname ASC)

AND bloom_filter_fp_chance = 0.01

AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

AND comment = ''

AND compaction = {'bucket_high': '1.5', 'bucket_low': '0.5', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'enabled': 'true', 'max_threshold': '32', 'min_sstable_size': '50', 'min_threshold': '4', 'tombstone_compaction_interval': '86400', 'tombstone_threshold': '0.2', 'unchecked_tombstone_compaction': 'false'}

AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}

AND crc_check_chance = 1.0

AND dclocal_read_repair_chance = 0.1

AND default_time_to_live = 0

AND gc_grace_seconds = 864000

AND max_index_interval = 2048

AND memtable_flush_period_in_ms = 0

AND min_index_interval = 128

AND read_repair_chance = 0.0

AND speculative_retry = '99PERCENTILE';

And the content of it looks like this

cqlsh:bth> select * from bth.employee;

 id | lastname    | firstname | dateofbirth
----+-------------+-----------+-------------
  1 |    eriksson |    tobias |  1973-06-11
  2 |  holgersson |    marcus |  1972-08-01
  7 |    eriksson |     linda |  1974-06-11
  3 | abrahamsson |    ingvar |  1959-10-05

(4 rows)
cqlsh:bth>

cqlsh:bth> select * from bth.employee;

id | lastname | firstname | dateofbirth

----+-------------+-----------+-------------

1 | eriksson | tobias | 1973-06-11

2 | holgersson | marcus | 1972-08-01

7 | eriksson | linda | 1974-06-11

3 | abrahamsson | ingvar | 1959-10-05

(4 rows)

cqlsh:bth>

And now I would like to search for all the rows that has a first name that starts with a ‘t’

In SQL that would have been :

SELECTÂ * FROMÂ bth.employee WHEREÂ firstname LIKEÂ ‘t%’;

In fact we could have done that on any column …. but in Cassandra it would result in something like this:

cqlsh:bth> SELECT * FROM bth.employee WHERE firstname like 't%';
InvalidRequest: code=2200 [Invalid query] message="firstname LIKE '<term>%' restriction is only supported on properly indexed columns"
cqlsh:bth>

cqlsh:bth> SELECT * FROM bth.employee WHERE firstname like 't%';

InvalidRequest: code=2200 [Invalid query] message="firstname LIKE '<term>%' restriction is only supported on properly indexed columns"

cqlsh:bth>

In Cassandra we first has to decide on which columns this should be possible, by creating an index like this:

CREATE CUSTOM INDEX employee_firstname_idx ON bth.employee (firstname) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer', 'case_sensitive': 'false'};

1	CREATE CUSTOM INDEX employee_firstname_idx ON bth.employee (firstname) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer', 'case_sensitive': 'false'};

And so you can now do the following

cqlsh:bth>SELECT * FROM bth.employee WHERE firstname LIKE 't%';

 id | lastname | firstname | dateofbirth
----+----------+-----------+-------------
  1 | eriksson |    tobias |  1973-06-11

(1 rows)
cqlsh:bth>

cqlsh:bth>SELECT * FROM bth.employee WHERE firstname LIKE 't%';

id | lastname | firstname | dateofbirth

----+----------+-----------+-------------

1 | eriksson | tobias | 1973-06-11

(1 rows)

cqlsh:bth>

But what if you decide that I would like to know all the employees that ends with an ‘s’ in their name, so something like this:

cqlsh:bth> SELECT * FROM bth.employee WHERE firstname like '%s';
InvalidRequest: code=2200 [Invalid query] message="firstname LIKE '%<term>' restriction is only supported on properly indexed columns"
cqlsh:bth>

cqlsh:bth> SELECT * FROM bth.employee WHERE firstname like '%s';

InvalidRequest: code=2200 [Invalid query] message="firstname LIKE '%<term>' restriction is only supported on properly indexed columns"

cqlsh:bth>

So to be able to search for something that contains we have to change the index like this instead:

CREATE CUSTOM INDEX employee_firstname_idx ON bth.employee (firstname) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {'mode': 'CONTAINS', 'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer', 'case_sensitive': 'false'};

1	CREATE CUSTOM INDEX employee_firstname_idx ON bth.employee (firstname) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {'mode': 'CONTAINS', 'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer', 'case_sensitive': 'false'};

And now you can run that query again:

cqlsh:bth> SELECT * FROM bth.employee WHERE firstname like '%s';

 id | lastname   | firstname | dateofbirth
----+------------+-----------+-------------
  1 |   eriksson |    tobias |  1973-06-11
  2 | holgersson |    marcus |  1972-08-01

(2 rows)
cqlsh:bth>

cqlsh:bth> SELECT * FROM bth.employee WHERE firstname like '%s';

id | lastname | firstname | dateofbirth

----+------------+-----------+-------------

1 | eriksson | tobias | 1973-06-11

2 | holgersson | marcus | 1972-08-01

(2 rows)

cqlsh:bth>

You can read more about the SASI index hereÂ https://docs.datastax.com/en/cql/3.3/cql/cql_reference/refCreateSASIIndex.html

Enjoy!

-Tobias

UDF/User Defined Functions in Cassandra 3.x

Posted on August 11, 2016 by tobias

I was just playing around with Cassandra WRITETIME and thought it was somewhat difficult to figure out the date / timestamp of a number like this (microseconds since EPOC) 1470645914253000.

So in my example it looked like this

cqlsh:bth> select id, writetime(dateofbirth) from bth.employee;

 id | writetime(dateofbirth)
----+------------------------
  1 |       1470645914253000
  2 |       1470645977177000
  7 |       1470948508799001
  3 |       1470645977178000

(4 rows)
cqlsh:bth>

cqlsh:bth> select id, writetime(dateofbirth) from bth.employee;

id | writetime(dateofbirth)

----+------------------------

1 | 1470645914253000

2 | 1470645977177000

7 | 1470948508799001

3 | 1470645977178000

(4 rows)

cqlsh:bth>

So I figured why not create a UDF that would solve this for me

That turned out to be a little bit of a challenge …

I thought that I could do like this

CREATE FUNCTION bth.ts2date ( input bigint )
	RETURNS NULL ON NULL INPUT
    RETURNS text 
    LANGUAGE java
    AS $$
    	if( input > 0L ) {
    		long ms = input / 1000L;
    		Date date=new Date(ms);
    		SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSS");
    		return sdf.format(date);
    	} else return null;
    $$;

CREATE FUNCTION bth.ts2date ( input bigint )

RETURNS NULL ON NULL INPUT

RETURNS text

LANGUAGE java

AS $$

if( input > 0L ) {

long ms = input / 1000L;

Date date=new Date(ms);

SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSS");

return sdf.format(date);

} else return null;

$$;

BUT NO, YOU CAN NOT!!!

There are several WRONGS in here it turns out

First off you have to turn on
enable_user_defined_functions: true
in the conf/cassandra.yaml file
All classes has to be fully qualified, so Date would be java.util.Date, and so on…
The division operator ‘/’ can not be used !!! however +,- and * works fine. surely this must be a bug … this called for some thinking…

The error I got when trying to use the code above without fully qualified names was

cqlsh> CREATE FUNCTION ts2date ( input bigint )
   ... RETURNS NULL ON NULL INPUT
   ...     RETURNS text 
   ...     LANGUAGE java
   ...     AS $$
   ...     if( input != null ) {
   ...     Date date=new Date(mills);
InvalidRequest: code=2200 [Invalid query] message="Functions must be fully qualified with a keyspace name if a keyspace is not set for the session"

cqlsh> CREATE FUNCTION ts2date ( input bigint )

... RETURNS NULL ON NULL INPUT

... RETURNS text

... LANGUAGE java

... AS $$

... if( input != null ) {

... Date date=new Date(mills);

InvalidRequest: code=2200 [Invalid query] message="Functions must be fully qualified with a keyspace name if a keyspace is not set for the session"

And the reason, if I got it right, is that you can not do imports.

The error I got when trying to use the division ‘/’ operator was this:

cqlsh:bth> CREATE FUNCTION bth.ts2date ( input bigint )
       ... RETURNS NULL ON NULL INPUT
       ...     RETURNS text 
       ...     LANGUAGE java
       ...     AS $$
       ...     if( input > 0L ) {
       ...     java.util.Date date=new java.util.Date(input/1000);
Invalid syntax at line 7, char 49
      java.util.Date date=new java.util.Date(input/1000);
                                                  ^
cqlsh:bth>

cqlsh:bth> CREATE FUNCTION bth.ts2date ( input bigint )

... RETURNS NULL ON NULL INPUT

... RETURNS text

... LANGUAGE java

... AS $$

... if( input > 0L ) {

... java.util.Date date=new java.util.Date(input/1000);

Invalid syntax at line 7, char 49

java.util.Date date=new java.util.Date(input/1000);

cqlsh:bth>

The code that works looks like this, using java.math.BigDecimal to solve it was perhaps a so-so solution, but it works:

CREATE FUNCTION bth.ts2date ( input bigint ) 
RETURNS NULL ON NULL INPUT     
RETURNS text      
LANGUAGE java     
AS $$     
	if( input > 0L ) {     
		java.math.BigDecimal t = java.math.BigDecimal.valueOf(1000L);     
		java.math.BigDecimal inp = java.math.BigDecimal.valueOf(input);     
		java.math.BigDecimal mst = inp.divide(t); 
		long ms = mst.longValue();     
		java.util.Date date=new java.util.Date(ms);     
		java.text.SimpleDateFormat sdf = new java.text.SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSS");     
		return sdf.format(date);     
	} else return null;     
$$;

CREATE FUNCTION bth.ts2date ( input bigint )

RETURNS NULL ON NULL INPUT

RETURNS text

LANGUAGE java

AS $$

if( input > 0L ) {

java.math.BigDecimal t = java.math.BigDecimal.valueOf(1000L);

java.math.BigDecimal inp = java.math.BigDecimal.valueOf(input);

java.math.BigDecimal mst = inp.divide(t);

long ms = mst.longValue();

java.util.Date date=new java.util.Date(ms);

java.text.SimpleDateFormat sdf = new java.text.SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSS");

return sdf.format(date);

} else return null;

$$;

So now my output in cqlsh.sh looks like this now

cqlsh:bth> select id, ts2date(writetime(dateofbirth)) from bth.employee;

 id | bth.ts2date(writetime(dateofbirth))
----+-------------------------------------
  1 |             2016-08-08 10:45:14,253
  2 |             2016-08-08 10:46:17,177
  7 |             2016-08-11 22:48:28,799
  3 |             2016-08-08 10:46:17,178

(4 rows)
cqlsh:bth>

cqlsh:bth> select id, ts2date(writetime(dateofbirth)) from bth.employee;

id | bth.ts2date(writetime(dateofbirth))

----+-------------------------------------

1 | 2016-08-08 10:45:14,253

2 | 2016-08-08 10:46:17,177

7 | 2016-08-11 22:48:28,799

3 | 2016-08-08 10:46:17,178

(4 rows)

cqlsh:bth>

That is a lot better !

Cassandra set the writetime explicitly with a PreparedStatement

Posted on March 16, 2016 by tobias

This is a quick one, I wanted to set the writetime of a row explicitly when I populate the database for testing purposes. We use the writetime of a column to filter them out.

It required some looking around to find out how to do this…. so I figured I write an article about it.

INSERT INTO invoices (invoice_id,amount,tax,description) VALUES ('333',10,3,'ice-cream') USING TIMESTAMP 1458134077121;

1	INSERT INTO invoices (invoice_id,amount,tax,description) VALUES ('333',10,3,'ice-cream') USING TIMESTAMP 1458134077121;

The timestamp will be set for ALL cells in this row (well not the primary key, cause it does not have a timestamp, but the others).

The timestamp is given as millisecondsÂ since EPOC, so lots of digitsÂ :-).

A prepared statement would then look like this (Scala code)

val cql = "INSERT INTO invoices (invoice_id,amount,tax,description) VALUES (?,?,?,?) USING TIMESTAMP ?;"

val stmt = session.prepare( cql )

val bs = stmt.bind()

bs.setString("invoice_id", "333" )

bs.setLong("amount", 10L )

bs.setLong("tax", 3L )

bs.setString("description","ice-cream")

bs.setLong("[timestamp]", 1458134077121L )

val result = Â session.execute( bs )

val cql = "INSERT INTO invoices (invoice_id,amount,tax,description) VALUES (?,?,?,?) USING TIMESTAMP ?;"

val stmt = session.prepare( cql )

val bs = stmt.bind()

bs.setString("invoice_id", "333" )

bs.setLong("amount", 10L )

bs.setLong("tax", 3L )

bs.setString("description","ice-cream")

bs.setLong("[timestamp]", 1458134077121L )

val result = Â session.execute( bs )

TTL and TIMESTAMP can both be set like this, i.e. with [ttl] and [timestamp]

-Tobias

Apache SPARK and Cassandra and SQL

Posted on January 21, 2016 by tobias

This is a short intro to start using Apache SPARK with Cassandra, running SQL on the Cassandra tables.

Note that I am not running a SPARK cluster, I am running “local”, to me this is really convenient, not having to run a SPARK server and workers for something so small. So for playing around with SPARK and Cassandra this is really good.

I am using Scala and SBT.

Something I was struggling hard with, to get the dependency versions right. It is crucial that you do not do like I did first, use version 1.5.2 of Spark and 1.5.0 for SparkCassandraConnector, this will NOT work. I constantly got exception withÂ java.lang.NoSuchMethodException, so incredibly frustrating to try out version after version.

Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.LogicalRelation.<init>(Lorg/apache/spark/sql/sources/BaseRelation;)V

1	Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.LogicalRelation.<init>(Lorg/apache/spark/sql/sources/BaseRelation;)V

build.sbt

val sparkV = "1.5.0"
val sparkCassandraConnectorV = "1.5.0-RC1"
val cassandraDriverV = "3.0.0-rc1"

libraryDependencies += "joda-time"           %   "joda-time"         % "2.9.1"
libraryDependencies += "org.slf4j"           %   "slf4j-api"         % "1.7.10"
libraryDependencies += "ch.qos.logback"      %   "logback-core"      % "1.1.3"
libraryDependencies += "ch.qos.logback"      %   "logback-classic"   % "1.1.3"
libraryDependencies += "jline"      %   "jline"   % "2.12.1"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % sparkV exclude("org.scala-lang","scala-compiler") exclude("jline","jline")
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % sparkV
libraryDependencies += "com.datastax.spark"  %% "spark-cassandra-connector" % sparkCassandraConnectorV exclude("org.apache.spark","spark-sql_2.11")
libraryDependencies += "com.datastax.cassandra" % "cassandra-driver-core" % cassandraDriverV

mainClass in (Compile, run) := Some("com.tsoft.dreamtel.spark.poc.SparkTest")

val sparkV = "1.5.0"

val sparkCassandraConnectorV = "1.5.0-RC1"

val cassandraDriverV = "3.0.0-rc1"

libraryDependencies += "joda-time" % "joda-time" % "2.9.1"

libraryDependencies += "org.slf4j" % "slf4j-api" % "1.7.10"

libraryDependencies += "ch.qos.logback" % "logback-core" % "1.1.3"

libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.3"

libraryDependencies += "jline" % "jline" % "2.12.1"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % sparkV exclude("org.scala-lang","scala-compiler") exclude("jline","jline")

libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % sparkV

libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % sparkCassandraConnectorV exclude("org.apache.spark","spark-sql_2.11")

libraryDependencies += "com.datastax.cassandra" % "cassandra-driver-core" % cassandraDriverV

mainClass in (Compile, run) := Some("com.tsoft.dreamtel.spark.poc.SparkTest")

A small Scala program to show how it works

SparkTest.scala

package com.tsoft.dreamtel.spark.poc

import org.apache.spark.sql.cassandra.CassandraSQLContext
import org.apache.spark.sql.types.{StringType, StructField, StructType}
import org.apache.spark.sql.{DataFrame, SQLContext}
import com.datastax.spark.connector._
import org.apache.spark.{SparkConf, SparkContext}

object SparkTest {
  def main(args: Array[String]): Unit = {
    testCassandraSQL()
  }

  def testCassandraSQL(): Unit = {
    val conf = new SparkConf(true).set("spark.cassandra.connection.host","127.0.0.1")
    val sc = new SparkContext("local", "Tobias-Local-SPARK-Context", conf)
    val cc = new CassandraSQLContext(sc)
    val sqlrdd: DataFrame = cc.sql("SELECT firstname from playground.individual WHERE firstname like 'L%'") //  WHERE firstname like 'S%'")
    sqlrdd.foreach( e => println("ROW: "+e.getString(0) ) )
  }


  }
}

package com.tsoft.dreamtel.spark.poc

import org.apache.spark.sql.cassandra.CassandraSQLContext

import org.apache.spark.sql.types.{StringType, StructField, StructType}

import org.apache.spark.sql.{DataFrame, SQLContext}

import com.datastax.spark.connector._

import org.apache.spark.{SparkConf, SparkContext}

object SparkTest {

def main(args: Array[String]): Unit = {

testCassandraSQL()

}

def testCassandraSQL(): Unit = {

val conf = new SparkConf(true).set("spark.cassandra.connection.host","127.0.0.1")

val sc = new SparkContext("local", "Tobias-Local-SPARK-Context", conf)

val cc = new CassandraSQLContext(sc)

val sqlrdd: DataFrame = cc.sql("SELECT firstname from playground.individual WHERE firstname like 'L%'") // WHERE firstname like 'S%'")

sqlrdd.foreach( e => println("ROW: "+e.getString(0) ) )

}

The output…

ROW: Leona
ROW: Lillian 
ROW: Liana
ROW: Lilian
ROW: Lovisa
ROW: Layla 
ROW: Lennon 
ROW: Leonidas 
ROW: Lova
ROW: LÃ©on
ROW: LÃ©on
ROW: Laila
ROW: Leonardo

ROW: Leona

ROW: Lillian

ROW: Liana

ROW: Lilian

ROW: Lovisa

ROW: Layla

ROW: Lennon

ROW: Leonidas

ROW: Lova

ROW: LÃ©on

ROW: Laila

ROW: Leonardo

SBT Good to know…

Posted on November 5, 2015 by tobias

Dependecy problems

I have been having some difficulties figuring out what depends on what. I found the following set plugins which I think can be really helpful;

https://github.com/jrudolph/sbt-dependency-graph

and

https://github.com/gilt/sbt-dependency-graph-sugar

Be sure to install GraphWiz first, I used Homebrew on my Mac

brew install graphviz

and I also had to create a config file

mkdir ~/.sbt/gilt
vi ~/.sbt/gilt/sbt-dependency-graph-sugar-cmd

1 2	mkdir ~/.sbt/gilt vi ~/.sbt/gilt/sbt-dependency-graph-sugar-cmd

with the following content

open -a Safari $1

1	open -a Safari $1

The readme explains how to use it pretty well, simply start sbtÂ CLI

Tobiass-MBP:projectX tobias$ sbt
[info] Loading global plugins from /Users/tobias/.sbt/0.13/plugins
[info] Updating {file:/Users/tobias/.sbt/0.13/plugins/}global-plugins...
Waiting for lock on /Users/tobias/.ivy2/.sbt.ivy.lock to be available...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Loading project definition from /Users/tobias/qvantel/projectX/project
Waiting for lock on /Users/tobias/.ivy2/.sbt.ivy.lock to be available...
[info] Set current project to projectX (in build file:/Users/tobias/qvantel/projectX/)
> dependencySvgView

Tobiass-MBP:projectX tobias$ sbt

[info] Loading global plugins from /Users/tobias/.sbt/0.13/plugins

[info] Updating {file:/Users/tobias/.sbt/0.13/plugins/}global-plugins...

Waiting for lock on /Users/tobias/.ivy2/.sbt.ivy.lock to be available...

[info] Resolving org.fusesource.jansi#jansi;1.4 ...

[info] Done updating.

[info] Loading project definition from /Users/tobias/qvantel/projectX/project

Waiting for lock on /Users/tobias/.ivy2/.sbt.ivy.lock to be available...

[info] Set current project to projectX (in build file:/Users/tobias/qvantel/projectX/)

> dependencySvgView

It will give you a graph that looks something like this (it is in SVG format so it is searchable!!!) Now you should see which package/jar is using which, and also where the different versions clash…

Show the class path for the run command

Tobiass-MBP:projectX tobias$ sbt
[info] Loading global plugins from /Users/tobias/.sbt/0.13/plugins
[info] Loading project definition from /Users/tobias/qvantel/projectX/project
[info] Set current project to projectX (in build file:/Users/tobias/qvantel/projectX/)
> show runtime:fullClasspath
[info] Updating {file:/Users/tobias/qvantel/projectX/}projectx...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[info] List(Attributed(/Users/tobias/qvantel/projectX/target/scala-2.11/classes), Attributed(/Users/tobias/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.7.jar), Attributed(/Users/tobias/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1.7.10.jar), Attributed(/Users/tobias/.ivy2/cache/ch.qos.logback/logback-core/jars/logback-core-1.1.3.jar), Attributed(/Users/tobias/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.3.jar), Attributed(/Users/tobias/.ivy2/cache/io.kamon/kamon-core_2.11/jars/kamon-core_2.11-0.5.2.jar), Attributed(/Users/tobias/.ivy2/cache/com.typesafe.akka/akka-actor_2.11/jars/akka-actor_2.11-2.3.14.jar), Attributed(/Users/tobias/.ivy2/cache/com.typesafe/config/bundles/config-1.2.1.jar), Attributed(/Users/tobias/.ivy2/cache/org.hdrhistogram/HdrHistogram/bundles/HdrHistogram-2.1.7.jar), Attributed(/Users/tobias/.ivy2/cache/io.kamon/kamon-akka_2.11/jars/kamon-akka_2.11-0.5.2.jar), Attributed(/Users/tobias/.ivy2/cache/io.kamon/kamon-scala_2.11/jars/kamon-scala_2.11-0.5.2.jar), Attributed(/Users/tobias/.ivy2/cache/io.kamon/kamon-spray_2.11/jars/kamon-spray_2.11-0.5.2.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-can_2.11/bundles/spray-can_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-io_2.11/bundles/spray-io_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-util_2.11/bundles/spray-util_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-http_2.11/bundles/spray-http_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/org.parboiled/parboiled-scala_2.11/jars/parboiled-scala_2.11-1.1.7.jar), Attributed(/Users/tobias/.ivy2/cache/org.parboiled/parboiled-core/jars/parboiled-core-1.1.7.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-client_2.11/bundles/spray-client_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-httpx_2.11/bundles/spray-httpx_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/org.scala-lang.modules/scala-xml_2.11/bundles/scala-xml_2.11-1.0.3.jar), Attributed(/Users/tobias/.ivy2/cache/org.jvnet.mimepull/mimepull/jars/mimepull-1.9.5.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-routing_2.11/bundles/spray-routing_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/com.chuusai/shapeless_2.11/jars/shapeless_2.11-1.2.4.jar), Attributed(/Users/tobias/.ivy2/cache/org.aspectj/aspectjweaver/jars/aspectjweaver-1.8.7.jar))
[success] Total time: 1 s, completed Nov 5, 2015 10:21:30 AM
> 
>

Tobiass-MBP:projectX tobias$ sbt

[info] Loading global plugins from /Users/tobias/.sbt/0.13/plugins

[info] Loading project definition from /Users/tobias/qvantel/projectX/project

[info] Set current project to projectX (in build file:/Users/tobias/qvantel/projectX/)

> show runtime:fullClasspath

[info] Updating {file:/Users/tobias/qvantel/projectX/}projectx...

[info] Resolving jline#jline;2.12.1 ...

[info] Done updating.

[info] List(Attributed(/Users/tobias/qvantel/projectX/target/scala-2.11/classes), Attributed(/Users/tobias/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.7.jar), Attributed(/Users/tobias/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1.7.10.jar), Attributed(/Users/tobias/.ivy2/cache/ch.qos.logback/logback-core/jars/logback-core-1.1.3.jar), Attributed(/Users/tobias/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.3.jar), Attributed(/Users/tobias/.ivy2/cache/io.kamon/kamon-core_2.11/jars/kamon-core_2.11-0.5.2.jar), Attributed(/Users/tobias/.ivy2/cache/com.typesafe.akka/akka-actor_2.11/jars/akka-actor_2.11-2.3.14.jar), Attributed(/Users/tobias/.ivy2/cache/com.typesafe/config/bundles/config-1.2.1.jar), Attributed(/Users/tobias/.ivy2/cache/org.hdrhistogram/HdrHistogram/bundles/HdrHistogram-2.1.7.jar), Attributed(/Users/tobias/.ivy2/cache/io.kamon/kamon-akka_2.11/jars/kamon-akka_2.11-0.5.2.jar), Attributed(/Users/tobias/.ivy2/cache/io.kamon/kamon-scala_2.11/jars/kamon-scala_2.11-0.5.2.jar), Attributed(/Users/tobias/.ivy2/cache/io.kamon/kamon-spray_2.11/jars/kamon-spray_2.11-0.5.2.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-can_2.11/bundles/spray-can_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-io_2.11/bundles/spray-io_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-util_2.11/bundles/spray-util_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-http_2.11/bundles/spray-http_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/org.parboiled/parboiled-scala_2.11/jars/parboiled-scala_2.11-1.1.7.jar), Attributed(/Users/tobias/.ivy2/cache/org.parboiled/parboiled-core/jars/parboiled-core-1.1.7.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-client_2.11/bundles/spray-client_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-httpx_2.11/bundles/spray-httpx_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/org.scala-lang.modules/scala-xml_2.11/bundles/scala-xml_2.11-1.0.3.jar), Attributed(/Users/tobias/.ivy2/cache/org.jvnet.mimepull/mimepull/jars/mimepull-1.9.5.jar), Attributed(/Users/tobias/.ivy2/cache/io.spray/spray-routing_2.11/bundles/spray-routing_2.11-1.3.3.jar), Attributed(/Users/tobias/.ivy2/cache/com.chuusai/shapeless_2.11/jars/shapeless_2.11-1.2.4.jar), Attributed(/Users/tobias/.ivy2/cache/org.aspectj/aspectjweaver/jars/aspectjweaver-1.8.7.jar))

[success] Total time: 1 s, completed Nov 5, 2015 10:21:30 AM

Create an MBean (JMX) in Scala

Posted on November 3, 2015 by tobias

Create the MBean like this

trait OrderTrackerMBean {
  def getOrderId : Int
  def setOrderId(orderId:Int)
}

class OrderTracker extends OrderTrackerMBean {
  var orderId = 0
  override def getOrderId: Int = orderId
  override def setOrderId(_orderId: Int)= {
    orderId = _orderId
  }

}

trait OrderTrackerMBean {

def getOrderId : Int

def setOrderId(orderId:Int)

}

class OrderTracker extends OrderTrackerMBean {

var orderId = 0

override def getOrderId: Int = orderId

override def setOrderId(_orderId: Int)= {

orderId = _orderId

}

NOTE, that the interface/trait must end with MBean in the name

And this is how you register your MBean

package com.tsoft.playingWithLogback

import java.lang.management.ManagementFactory
import javax.management.{ObjectName, MBeanServer}

import org.slf4j.{MarkerFactory, Marker, LoggerFactory}
import org.slf4j.MDC
import scala.concurrent.{Await, Future}
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global

object LotsOfLogOutput {

  val logger = LoggerFactory.getLogger(this.getClass)

  val orderTrackerBean = new OrderTracker
  val mbs:MBeanServer = ManagementFactory.getPlatformMBeanServer();
  val mBeanName:ObjectName = new ObjectName("com.tsoft.playingWithLogback:type=Tracking");
  mbs.registerMBean( orderTrackerBean, mBeanName);

...

package com.tsoft.playingWithLogback

import java.lang.management.ManagementFactory

import javax.management.{ObjectName, MBeanServer}

import org.slf4j.{MarkerFactory, Marker, LoggerFactory}

import org.slf4j.MDC

import scala.concurrent.{Await, Future}

import scala.concurrent.duration._

import scala.concurrent.ExecutionContext.Implicits.global

object LotsOfLogOutput {

val logger = LoggerFactory.getLogger(this.getClass)

val orderTrackerBean = new OrderTracker

val mbs:MBeanServer = ManagementFactory.getPlatformMBeanServer();

val mBeanName:ObjectName = new ObjectName("com.tsoft.playingWithLogback:type=Tracking");

mbs.registerMBean( orderTrackerBean, mBeanName);

...

And the simply launch Java Mission Control (imc), attach to the JVM, and modify your MBean attributes as you like.

Enjoy!

SBT module not found, why ?

Posted on November 3, 2015 by tobias

I have an build.sbt file that looks like this

name := "TobiasPlayground"

version := "1.0"

scalaVersion := "2.11.7"

scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8")

libraryDependencies ++= {
  val akkaV = "2.3.9"
  val sprayV = "1.3.3"
  Seq(
    "io.spray"            %%  "spray-client"  % sprayV,
    "io.spray"            %%  "spray-can"     % sprayV,
    "io.spray"            %%  "spray-routing" % sprayV,
    "io.spray"            %%  "spray-testkit" % sprayV  % "test",
    "com.typesafe.akka"   %%  "akka-actor"    % akkaV,
    "com.typesafe.akka"   %%  "akka-testkit"  % akkaV   % "test",
    "org.specs2"          %%  "specs2-core"   % "2.3.11" % "test",
    "org.slf4j"           %%   "slf4j-api"      % "1.7.10"
  )
}

name := "TobiasPlayground"

version := "1.0"

scalaVersion := "2.11.7"

scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8")

libraryDependencies ++= {

val akkaV = "2.3.9"

val sprayV = "1.3.3"

Seq(

"io.spray" %% "spray-client" % sprayV,

"io.spray" %% "spray-can" % sprayV,

"io.spray" %% "spray-routing" % sprayV,

"io.spray" %% "spray-testkit" % sprayV % "test",

"com.typesafe.akka" %% "akka-actor" % akkaV,

"com.typesafe.akka" %% "akka-testkit" % akkaV % "test",

"org.specs2" %% "specs2-core" % "2.3.11" % "test",

"org.slf4j" %% "slf4j-api" % "1.7.10"

)

}

But for some reason I can’t get slf4j downloaded from the Maven repository (http://mvnrepository.com)

If I search the Maven Repository, I can clearly see that the version I intend to use is there.

Running “sbt compile” from the command line will result in the following output, and here it is time to pay attention to the details, look at what it is trying to do !!!

Tobiass-MBP:TobiasPlayground tobias$ sbt compile
[info] Loading project definition from /Users/tobias/qvantel/TobiasPlayground/project
Waiting for lock on /Users/tobias/.ivy2/.sbt.ivy.lock to be available...
[info] Set current project to TobiasPlayground (in build file:/Users/tobias/qvantel/TobiasPlayground/)
[info] Updating {file:/Users/tobias/qvantel/TobiasPlayground/}tobiasplayground...
Waiting for lock on /Users/tobias/.ivy2/.sbt.ivy.lock to be available...
[info] Resolving org.slf4j#slf4j-api_2.11;1.7.10 ...
[warn] 	module not found: org.slf4j#slf4j-api_2.11;1.7.10
[warn] ==== local: tried
[warn]   /Users/tobias/.ivy2/local/org.slf4j/slf4j-api_2.11/1.7.10/ivys/ivy.xml
[warn] ==== jcenter: tried
[warn]   https://jcenter.bintray.com/org/slf4j/slf4j-api_2.11/1.7.10/slf4j-api_2.11-1.7.10.pom
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/org/slf4j/slf4j-api_2.11/1.7.10/slf4j-api_2.11-1.7.10.pom
[info] Resolving jline#jline;2.12.1 ...
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::          UNRESOLVED DEPENDENCIES         ::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	:: org.slf4j#slf4j-api_2.11;1.7.10: not found
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 
[warn] 	Note: Unresolved dependencies path:
[warn] 		org.slf4j:slf4j-api_2.11:1.7.10 (/Users/tobias/qvantel/TobiasPlayground/build.sbt#L9-27)
[warn] 		  +- default:tobiasplayground_2.11:1.0

Tobiass-MBP:TobiasPlayground tobias$ sbt compile

[info] Loading project definition from /Users/tobias/qvantel/TobiasPlayground/project

Waiting for lock on /Users/tobias/.ivy2/.sbt.ivy.lock to be available...

[info] Set current project to TobiasPlayground (in build file:/Users/tobias/qvantel/TobiasPlayground/)

[info] Updating {file:/Users/tobias/qvantel/TobiasPlayground/}tobiasplayground...

Waiting for lock on /Users/tobias/.ivy2/.sbt.ivy.lock to be available...

[info] Resolving org.slf4j#slf4j-api_2.11;1.7.10 ...

[warn] module not found: org.slf4j#slf4j-api_2.11;1.7.10

[warn] ==== local: tried

[warn] /Users/tobias/.ivy2/local/org.slf4j/slf4j-api_2.11/1.7.10/ivys/ivy.xml

[warn] ==== jcenter: tried

[warn] https://jcenter.bintray.com/org/slf4j/slf4j-api_2.11/1.7.10/slf4j-api_2.11-1.7.10.pom

[warn] ==== public: tried

[warn] https://repo1.maven.org/maven2/org/slf4j/slf4j-api_2.11/1.7.10/slf4j-api_2.11-1.7.10.pom

[info] Resolving jline#jline;2.12.1 ...

[warn] ::::::::::::::::::::::::::::::::::::::::::::::

[warn] :: UNRESOLVED DEPENDENCIES ::

[warn] ::::::::::::::::::::::::::::::::::::::::::::::

[warn] :: org.slf4j#slf4j-api_2.11;1.7.10: not found

[warn] ::::::::::::::::::::::::::::::::::::::::::::::

[warn]

[warn] Note: Unresolved dependencies path:

[warn] org.slf4j:slf4j-api_2.11:1.7.10 (/Users/tobias/qvantel/TobiasPlayground/build.sbt#L9-27)

[warn] +- default:tobiasplayground_2.11:1.0

As you can see above the package (jar) it tries to download is not slf4j-api it is slf4j-api_2.11;1.7.10

The build.sbt file uses the double and single % (percent) character and this is what makes the difference. Â The %% makes SBT append the project specified scala version to the package name, resulting in a name “slf4j-api_2.11”.

BUT that name does not exist in the Maven Repository, however, the “slf4j-api” does

So by simply choosing one instead of two %, the problem will go away 🙂

Thus, the built.sbt should look like this instead

name := "TobiasPlayground"

version := "1.0"

scalaVersion := "2.11.7"

scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8")

libraryDependencies ++= {
  val akkaV = "2.3.9"
  val sprayV = "1.3.3"
  Seq(
    "io.spray"            %%  "spray-client"  % sprayV,
    "io.spray"            %%  "spray-can"     % sprayV,
    "io.spray"            %%  "spray-routing" % sprayV,
    "io.spray"            %%  "spray-testkit" % sprayV  % "test",
    "com.typesafe.akka"   %%  "akka-actor"    % akkaV,
    "com.typesafe.akka"   %%  "akka-testkit"  % akkaV   % "test",
    "org.specs2"          %%  "specs2-core"   % "2.3.11" % "test",
    "org.slf4j"           %   "slf4j-api"      % "1.7.10"
  )
}

name := "TobiasPlayground"

version := "1.0"

scalaVersion := "2.11.7"

scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8")

libraryDependencies ++= {

val akkaV = "2.3.9"

val sprayV = "1.3.3"

Seq(

"io.spray" %% "spray-client" % sprayV,

"io.spray" %% "spray-can" % sprayV,

"io.spray" %% "spray-routing" % sprayV,

"io.spray" %% "spray-testkit" % sprayV % "test",

"com.typesafe.akka" %% "akka-actor" % akkaV,

"com.typesafe.akka" %% "akka-testkit" % akkaV % "test",

"org.specs2" %% "specs2-core" % "2.3.11" % "test",

"org.slf4j" % "slf4j-api" % "1.7.10"

)

}

For reference go toÂ https://www.playframework.com/documentation/2.1.1/SBTDependenciesÂ to read more about how SBT handles decencies and the % and %%.

I read this article to finally get this rightÂ http://stackoverflow.com/questions/17461453/build-scala-and-symbols-meaning

See who is connected to the same network / WIFI

Posted on October 11, 2015 by tobias

There are tools out there, such as Angry IP Scanner, iNet or IP Scanner, Â but this can easily be done using a bash script, not as nice looking perhaps but it does the job.

The script :

#!/bin/bash
for i in {0..255}
do
	cmd="nslookup 192.168.1."
	cmd=$cmd$i
	result=$($cmd|egrep "Address|name")
	echo "****************"
	echo $cmd
	echo $result
done

#!/bin/bash

for i in {0..255}

cmd="nslookup 192.168.1."

cmd=$cmd$i

result=$($cmd|egrep "Address|name")

echo "****************"

echo $cmd

echo $result

done

Which should give you an output that looks like this

****************
nslookup 192.168.1.84
Address: 192.168.1.254#53
****************
nslookup 192.168.1.85
Address: 192.168.1.254#53
****************
nslookup 192.168.1.86
Address: 192.168.1.254#53 86.1.168.192.in-addr.arpa name = Tobiass-iMac.lan.
****************
nslookup 192.168.1.87
Address: 192.168.1.254#53 87.1.168.192.in-addr.arpa name = raspberrypi.lan.
****************
nslookup 192.168.1.88
Address: 192.168.1.254#53 88.1.168.192.in-addr.arpa name = Lindas-iPad.lan.
****************
nslookup 192.168.1.89
Address: 192.168.1.254#53
****************

****************

nslookup 192.168.1.84

Address: 192.168.1.254#53

****************

nslookup 192.168.1.85

Address: 192.168.1.254#53

****************

nslookup 192.168.1.86

Address: 192.168.1.254#53 86.1.168.192.in-addr.arpa name = Tobiass-iMac.lan.

****************

nslookup 192.168.1.87

Address: 192.168.1.254#53 87.1.168.192.in-addr.arpa name = raspberrypi.lan.

****************

nslookup 192.168.1.88

Address: 192.168.1.254#53 88.1.168.192.in-addr.arpa name = Lindas-iPad.lan.

****************

nslookup 192.168.1.89

Address: 192.168.1.254#53

****************

That’s it

-Tobias

Ã–vervakning av min IVT-490 vÃ¤rmepump med Raspberry Pi

Posted on March 8, 2015 by tobias

I denna artikeln ska jag fÃ¶rsÃ¶ka beskriva mitt lilla projekt med att Ã¶vervaka min IVT-490 vÃ¤rmepump med en Raspberry P.

Jag har hÃ¤mtat mycket kunskap frÃ¥n dessa tvÃ¥ forum-trÃ¥dar :
http://www.varmepumpsforum.com/vpforum/index.php?topic=27306.15Tanken
och
http://www.byggahus.se/forum/varme-allmant/44487-prata-med-ivt-490-a-5.html

Min vÃ¤rmepump har ett interface som ser ut sÃ¥hÃ¤r :

Man behÃ¶ver slÃ¥ pÃ¥ variabeldumpning pÃ¥ pannan fÃ¶r att fÃ¥ ut information. Man kommer in i Admin mode genom att hÃ¥lla in hÃ¶gra knappen i 5-10 sekunder. Sedan letar man upp “Installation och igÃ¥ngkÃ¶rning” och dÃ¤r finner man “Variabeldumpning till PC”. Med det pÃ¥slaget sÃ¥ fÃ¥r man ut data 1 gÃ¥ng per minut. (source; Niclas Hedlund)

CiterarÂ Niclas.HedlundÂ om interfacet frÃ¥n VÃ¤rmepumpsforum:

Baudraten Ã¤r 9600baud och 8 databitar, 1stopbit och ingen paritet.
TÃ¤nk pÃ¥ att TxD som Ã¤r markerat pÃ¥ bilden Ã¤r data UT frÃ¥n pumpen, och RxD data IN till pumpen.

DÃ¥ jag inte lyckats fÃ¥ tag pÃ¥ en kontakt sÃ¥ fick jag anvÃ¤nda mig av en plastpÃ¥seklÃ¤mma (som man kÃ¶per pÃ¥ MAXI) som jag vÃ¤nt pÃ¥ och satt tvÃ¥ skruvar i, och sÃ¥ har jag silvertejp ocksÃ¥…. ja ja det Ã¤r ju pÃ¥ egen risk 🙂

EDIT: Nu funkade inte min plastklÃ¤mma sÃ¥ bra sÃ¥ jag lÃ¶dde dit trÃ¥darna istÃ¤llet, nu funkar det bÃ¤ttre. MEN sÃ¥ hÃ¶rde Fredrik Winkvist av sig och berÃ¤ttade vilken kontakt man skulle ha, och den ska funka perfekt. FÃ¶r er som sÃ¶ker sÃ¥ heter den “2.54mm Pitch 2×5 Pin Double Row Angle IDC Pin Headers 10 Pins”. Ett stort Tack till Fredrik, hÃ¤r Ã¤r en lÃ¤nk till eBay som sÃ¤kert blir outdated men sÃ¥ lÃ¤nge den finns kvar kan lÃ¤nkenÂ vara bra:Â http://www.ebay.com/itm/20-Pcs-2-54mm-Pitch-2×5-Pin-Double-Row-Angle-IDC-Pin-Headers-10-Pins-/310784076520?pt=LH_DefaultDomain_0&hash=item485c2caae8

Specification:

Product Name	IDC Pin Header
Position	5
Pin Number	10
Row	2
Mounting Angle	90 Degree
Pin Pitch	2.54mm/0.1″
Pin Length	4mm/ 0.16″Â Â Â Â 12mm/ 0.47″
Total Size	19 x 9.5 x 22mm/ 0.75″ x 0.37″ x 0.87″ ( LWH)
Material	Plastic, Metal
Color	Black, Silver Tone
Net Weight	66g
Package Content	20 x Angle IDC Pin Headers

Description:

Features double row, 90 degree mounting angle, male pin header, electronic component, IDC connector.

This pin header is an ideal connector for the PCB board and intergrated circuit, used widely in the computer, breadboard, LCD, TV and other home appliances.

NÃ¤r detta var gjort var jag tvungen att koppla in det via typ USB, sÃ¥ jag kÃ¶pte en sÃ¥dan

hÃ¤rÂ pÃ¥ ebay.Â http://www.ebay.co.uk/itm/6PIN-CP2102-USB-2-0-to-UART-TTL-Electronic-Connector-Serial-Converter-Module-/390568169609?pt=UK_Computing_Parallel_Serial_PS_2&hash=item5aefad5089
Varje minut sÃ¥ skickar den ut 37-parametrar som Ã¤r ;-separerade, se EXCEL filen;Â Loggning-2. (loggfilen kommer ocksÃ¥ frÃ¥n VÃ¤rmepumpsforumet; inskickat avÂ Larsmaja)

En rad kan se ut sÃ¥hÃ¤r :

360; Â  315;Â  Â  33; Â  478; Â  445; Â  447;Â  -512; Â  774;Â  -512;0;0;0;0;1;0;0;1;1;0;0; Â  296; Â  311; Â  326; Â  341; Â  495; Â  520; Â  545; Â  301; Â  326; Â  351; Â  435; Â  105;Â  Â  26; Â  Â  0; Â  Â  0; Â  Â  0; Â  Â  0

1	360; Â 315;Â Â 33; Â 478; Â 445; Â 447;Â -512; Â 774;Â -512;0;0;0;0;1;0;0;1;1;0;0; Â 296; Â 311; Â 326; Â 341; Â 495; Â 520; Â 545; Â 301; Â 326; Â 351; Â 435; Â 105;Â Â 26; Â Â 0; Â Â 0; Â Â 0; Â Â 0

DÃ¤r alla temperaturer Ã¤r heltal med 1 decimals noggrannhet, sÃ¥ 315 betyder 31,5 grader Celsius.
360 Ã¤r sekvensummer
315 Ã¤r Framledningstemperatur (GT1), dvs 31,5 grader
33 Ã¤r Utetemperatur (GT2), dvs 3,1 grader
478 Ã¤r Tapvarmvatten (GT3-1), dvs 47,8 grader
445 Ã¤r Varmvatten(GT3-2), dvs 44,5 grader
447 Ã¤r VÃ¤rmevatten(GT3-3), dvs 44,7 grader
-512 Ã¤r Rumstemperatur(GT5), men jag har inte den givaren sÃ¥ den visar -512
774 Ã¤r Hetgastemperatur(GT6), dvs 77,4 grader

I bilden nedan ser man de olika givarna GT1-7.

PÃ¥ min Raspberry pi skrev jag ett enkelt script som lÃ¤ser frÃ¥n USB-serieporten och postar pÃ¥ en websida som jag har gjort, dÃ¤r jag helt enkelt lÃ¤gger in det i en MySQL databas )oerhÃ¶rt enkelt, bara timestamp och sÃ¥ det ;-separerade datan (tvÃ¥ kolumner, pga av att jag ville komma igÃ¥ng, skulle sÃ¥klart haft 38 kolumner sÃ¥ man kunde gÃ¶ra lite mer SQL, men det fÃ¥r bli en annan gÃ¥ng…)

import requests
import serial
import time
from time import gmtime, strftime
import re

pattern = re.compile(r'\s+')

def readlineCR(port):
	rv = ""
	while True:
		ch = port.read()
		rv += ch
		if ch == '\r' or ch == '':
			return rv

def post(timestamp,data) :
	payload = {'timestamp': timestamp, 'data': data}
	try :
		r = requests.post("http://domain.com/path/file.php", data=payload)
		print(r.text)
	except Exception as e:
		print "Oops! unable to post result!",e.__doc__,e.message


port = serial.Serial("/dev/ttyUSB0", baudrate = 9600, timeout = 60)

while True: 
	rcv=''
	try :
		rcv = readlineCR(port)
	except Exception as e:
		print "Oops! problems reading from USB!"
		print e.__doc__
		print e.message
	rcv = re.sub(pattern, '',rcv)
	timeStamp = strftime("%Y-%m-%d %H:%M:%S", gmtime())
	print(timeStamp+rcv)
	if rcv :
		post( timeStamp,rcv)
	else :
		print "Empty data..."

import requests

import serial

import time

from time import gmtime, strftime

import re

pattern = re.compile(r'\s+')

def readlineCR(port):

rv = ""

while True:

ch = port.read()

rv += ch

if ch == '\r' or ch == '':

return rv

def post(timestamp,data) :

payload = {'timestamp': timestamp, 'data': data}

try :

r = requests.post("http://domain.com/path/file.php", data=payload)

print(r.text)

except Exception as e:

print "Oops! unable to post result!",e.__doc__,e.message

port = serial.Serial("/dev/ttyUSB0", baudrate = 9600, timeout = 60)

while True:

rcv=''

try :

rcv = readlineCR(port)

except Exception as e:

print "Oops! problems reading from USB!"

print e.__doc__

print e.message

rcv = re.sub(pattern, '',rcv)

timeStamp = strftime("%Y-%m-%d %H:%M:%S", gmtime())

print(timeStamp+rcv)

if rcv :

post( timeStamp,rcv)

else :

print "Empty data..."

Jag har noterat att ibland sÃ¥ ramlar mina trÃ¥dar ur, och sÃ¥ fÃ¥r jag inget resultat, dÃ¥ blir serieporten inte nÃ¥barÂ 🙂 och dÃ¥ fÃ¥r jag lite roliga felutskrifter… och sÃ¥klart blir diagrammen dÃ¥ligt uppdaterade….

Jag har gjort en websida (PHP and Javascript) dÃ¤r jag anvÃ¤nt Highcharrts fÃ¶r att rendera diagrammen. Ta en titt hÃ¤rÂ http://tsoft.se/varmepanna/

Fredrik Winkvist var dessutom lite uppfinningsrik och berÃ¤ttade att man anvÃ¤nda strÃ¶mfÃ¶rsÃ¶rjningen frÃ¥n vÃ¤rmepumpens 5Volt ut till Raspberry Pi;n. Ta en kik pÃ¥ bilden nedan.

FÃ¶rbÃ¤ttringsfÃ¶rslag Ã¤r alltid vÃ¤lkomna, sÃ¥ hÃ¶r av er !

Mitt nÃ¤sta projekt, nÃ¤r jag fÃ¥r tid…. Ã¤r att mÃ¤ta strÃ¶mfÃ¶rbrukningen hemma…

-Tobias

tsoft.se

Tobias – With a Passion For Software Development

Author Archives: tobias

Remove the cardo-updater agent from OSX

SQL LIKE operation in Cassandra, is possible in v3.4+

UDF/User Defined Functions in Cassandra 3.x

Cassandra set the writetime explicitly with a PreparedStatement

Apache SPARK and Cassandra and SQL

SBT Good to know…

Dependecy problems

Show the class path for the run command

Create an MBean (JMX) in Scala

SBT module not found, why ?

See who is connected to the same network / WIFI

Ã–vervakning av min IVT-490 vÃ¤rmepump med Raspberry Pi