Cygnus does not contain MySql database data
So, I've read all the documentation and followed the tutorial on MySQL Persistence , but I still can't seem to present any data in a MySQL database.
Even though I set the mode to presistence = row, it does not create any database or table.
What am I doing wrong?
My subscription:
python2.7 SetSubscription.py bustest4 http://localhost:5050/notify
Output:
* Asking to http://localhost:1026/v1/subscribeContext
* Headers: {'Fiware-Service': 'fiwaretestapi', 'content-type': 'application/json ', 'accept': 'application/json', 'X-Auth-Token': 'NULL'}
* Sending PAYLOAD:
{
"reference": "http://localhost:5050/notify",
"throttling": "PT5S",
"entities": [
{
"type": "",
"id": "bustest4",
"isPattern": "false"
}
],
"attributes": [
"temperature"
],
"duration": "P1M",
"notifyConditions": [
{
"condValues": [
"temperature"
],
"type": "ONCHANGE"
}
]
}
...
* Status Code: 200
{
"subscribeResponse" : {
"subscriptionId" : "5567332298add18cc3e183ac",
"duration" : "P1M",
"throttling" : "PT5S"
}
}
My agent_a1.conf:
#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel ckan-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = fiwaretestapi
# Default service path (service path semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service_path = /
# Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts de
# Timestamp interceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# Destination extractor interceptor, do not change
cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder
# Matching table for the destination extractor interceptor, put the right absolute path to the file if necessary
# See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.de.matching_table = /usr/cygnus/conf/matching_table.conf
# ============================================
# OrionMySQLSink configuration
# channel name from where to read notification events
cygnusagent.sinks.mysql-sink.channel = mysql-channel
# sink class, must not be changed
cygnusagent.sinks.mysql-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink
# the FQDN/IP address where the MySQL server runs
cygnusagent.sinks.mysql-sink.mysql_host = localhost
# the port where the MySQL server listes for incomming connections
cygnusagent.sinks.mysql-sink.mysql_port = 3306
# a valid user in the MySQL server
cygnusagent.sinks.mysql-sink.mysql_username = root
# password for the user above
cygnusagent.sinks.mysql-sink.mysql_password = ***********
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.mysql-sink.attr_persistence = row
#=============================================
# mysql-channel configuration
# channel type (must not be changed)
cygnusagent.channels.mysql-channel.type = memory
# capacity of the channel
cygnusagent.channels.mysql-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.mysql-channel.transactionCapacity = 100
My cygnus_instance_c1.conf:
# Who to run cygnus as. Note that you may need to use root if you want
# to run cygnus in a privileged port (<1024)
CYGNUS_USER=root
# Where is the config folder
CONFIG_FOLDER=/usr/cygnus/conf
# Which is the config file
CONFIG_FILE=/usr/cygnus/conf/agent_a1.conf
# Name of the agent. The name of the agent is not trivial, since it is the base for the Flume parameters
# naming conventions, e.g. it appears in .sources.http-source.channels=...
AGENT_NAME=cygnususer
# Name of the logfile located at /var/log/cygnus. It is important to put the extension '.log' in order to the log rotation works properly
LOGFILE_NAME=cygnus.log
# Administration port. Must be unique per instance
ADMIN_PORT=8081
My cygnus.log:
Info: Sourcing environment configuration script /usr/cygnus/conf/flume-env.sh
Warning: JAVA_HOME is not set!
+ exec /usr/bin/java -Xmx20m -Dflume.log.file=cygnus.log -cp '/usr/cygnus/conf:/usr/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/libext/*' -Djava.library.path= es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication -p 8081 -f /usr/cygnus/conf/agent_a1.conf -n root
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.7.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
EDIT:
So after some changes I got the log file and I found out that port 8081 is already in use. Can you explain to me what ADMIN_PORT is used for, which port is recommended?
Log file:
02 Jun 2015 05:16:40,680 INFO [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133) - Reloading configuration file:/usr/cygnus/conf/agent_a1.conf
02 Jun 2015 05:16:40,685 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,686 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,686 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930) - Added sinks: mysql-sink Agent: cygnusagent
02 Jun 2015 05:16:40,686 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,692 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140) - Post-validation flume configuration contains configuration for agents: [cygnusagent]
02 Jun 2015 05:16:40,692 WARN [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:138) - No configuration found for this host:root
02 Jun 2015 05:16:40,693 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101) - Shutting down configuration: { sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,693 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,694 INFO [conf-file-poller-0] (es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication.startManagementInterface:85) - Starting a Jetty server listening on port 8081 (Management Interface)
02 Jun 2015 05:16:40,695 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101) - Shutting down configuration: { sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,695 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,695 INFO [conf-file-poller-0] (es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication.startManagementInterface:85) - Starting a Jetty server listening on port 8081 (Management Interface)
02 Jun 2015 05:16:40,696 INFO [Thread-26] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
02 Jun 2015 05:16:40,704 WARN [Thread-26] (org.mortbay.log.Slf4jLog.warn:76) - failed SocketConnector@0.0.0.0:8081: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,704 WARN [Thread-26] (org.mortbay.log.Slf4jLog.warn:76) - failed Server@4f1b95f: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,704 FATAL [Thread-26] (es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer.run:63) - Fatal error running the Management Interface. Details=Address already in use
02 Jun 2015 05:16:40,705 INFO [Thread-27] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
02 Jun 2015 05:16:40,709 WARN [Thread-27] (org.mortbay.log.Slf4jLog.warn:76) - failed SocketConnector@0.0.0.0:8081: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,709 WARN [Thread-27] (org.mortbay.log.Slf4jLog.warn:76) - failed Server@ed4c222: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,709 FATAL [Thread-27] (es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer.run:63) - Fatal error running the Management Interface. Details=Address already in use
EDIT 2:
My script that updates the object in the Context Broker:
BASE_URL = 'http://localhost:1026'
UPDATE_URL = BASE_URL+'/ngsi10/updateContext'
HEADERS = {
'Content-Type': 'application/json',
'Accept': 'application/json',
'Fiware-Service' : 'fiwaretestapi',
'Fiware-ServicePath': '/'
}
UPDATE_EXAMPLE = {
"contextElements": [
{
"type": "",
"isPattern": "false",
"id": "bustest4",
"attributes": [
{
"name": "temperature",
"type": "int",
"value": "99"
}
]
}
],
"updateAction": "APPEND"
}
def post(url, data):
""""""
req = urllib2.Request(url, data, HEADERS)
f = urllib2.urlopen(req)
result = json.loads(f.read())
f.close()
return result
if __name__ == "__main__":
print post(UPDATE_URL, json.dumps(UPDATE_EXAMPLE))
EDIT3:
Even if I set admin port 8085 in the cygnus agent config, it tries to bind to 8081, is that ok?
Here are the logs from cygnus:
time=2015-06-12T05:56:39.820EDT | lvl=INFO | trans= | function=start | comp=Cygnu s | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component ty pe: CHANNEL, name: mysql-channel started
time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startAllComponents | comp=Cygnus | msg=org.apache.flume.node.Application[173] : Starting Sink mysql -sink
time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startAllComponents | comp=Cygnus | msg=org.apache.flume.node.Application[184] : Starting Source htt p-source
time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startManagementInt erface | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusAppl ication[85] : Starting a Jetty server listening on port 8081 (Management Interfac e)
time=2015-06-12T05:56:39.822EDT | lvl=INFO | trans= | function=start | comp=Cygnu s | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink[151] : [mysql- sink] Startup completed
time=2015-06-12T05:56:39.823EDT | lvl=INFO | trans= | function=info | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[67] : jetty-6.1.26
time=2015-06-12T05:56:39.824EDT | lvl=INFO | trans= | function=info | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[67] : jetty-6.1.26
time=2015-06-12T05:56:39.825EDT | lvl=INFO | trans= | function=info | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[67] : Started SocketConnector@0.0.0.0:5050
time=2015-06-12T05:56:39.825EDT | lvl=INFO | trans= | function=start | comp=Cygnu s | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component ty pe: SOURCE, name: http-source started
time=2015-06-12T05:56:39.827EDT | lvl=WARN | trans= | function=warn | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[76] : failed SocketConnector@0.0.0.0:8081: java.n et.BindException: Address already in use
time=2015-06-12T05:56:39.827EDT | lvl=WARN | trans= | function=warn | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[76] : failed Server@1c9c5521: java.net.BindExcept ion: Address already in use
time=2015-06-12T05:56:39.827EDT | lvl=FATAL | trans= | function=run | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer[63] : Fatal error r unning the Management Interface. Details=Address already in use
Sign up when I sign up:
time=2015-06-12T06:03:56.529EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle rs.OrionRestHandler[153] : Starting transaction (1434103313-190-0000000000)
time=2015-06-12T06:03:56.535EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle rs.OrionRestHandler[239] : Received data ({ "subscriptionId" : "557aae8c98add18c c3e183b6", "originator" : "localhost", "contextResponses" : [ { "contex tElement" : { "type" : "thing", "isPattern" : "false", "id" : "autocarro1", "attributes" : [ { "name" : "temperatu re", "type" : "int", "value" : "95", "metadatas" : [ { "name" : "TimeInstant", "type" : "ISO8601", "value" : "2015-06-03T09:17:44.046583Z" } ] } ] }, "statusCode" : { "code" : "200", "reasonPhrase" : "OK" } } ]})
time=2015-06-12T06:03:56.540EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle rs.OrionRestHandler[261] : Event put in the channel (id=1983722072, ttl=10)
time=2015-06-12T06:03:56.724EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or ionSink[126] : Event got from the channel (id=1983722072, headers={timestamp=1434 103436542, content-type=application/json, transactionId=1434103313-190-0000000000 , fiware-service=fiwaretestapi, fiware-servicepath=, ttl=10, destination=autocarr o1_thing}, bodyLength=657)
time=2015-06-12T06:03:57.260EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=persist | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or ionMySQLSink[227] : [mysql-sink] Persisting data at OrionMySQLSink. Database: fiw aretestapi, Table: autocarro1_thing, Data: 1434103436,2015-06-12T06:03:56.542,aut ocarro1,thing,temperature,thing,95,[{"name":"TimeInstant","type":"ISO8601","value ":"2015-06-03T09:17:44.046583Z"}]
time=2015-06-12T06:03:57.270EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or ionSink[187] : Finishing transaction (1434103313-190-0000000000)
+3
source to share
1 answer
As I can see in the log, it is most likely Cygnus is running, but not starting any Flume component (any source, channel, or sink). This is due to some configuration errors.
File-related agent_a1.conf
:
- There is no list of sources, channels and receivers:
cygnusagent.sources = http-source cygnusagent.sinks = mysql-sink cygnusagent.channels = mysql-channel
-
cygnusagent.sources.http-source.channels
The value should bemysql-channel
Concerning cygnus_instance_c1.conf
:
-
AGENT_NAME
The value should becygnusagent
Which version have you installed? Do you love Swan as a service or as an independent process?
Additionally, can you try to start Cygnus in DEBUG mode? Just edit the file /usr/cygnus/conf/log4j.properties
.
Make the suggested changes and see how the log changes! :)
EDIT 1
Such a "fatal mistake" is not so fatal. This was a bug introduced in Cygnus 0.7.1 and has now been fixed. Anyway, even in 0.7.1 this did not affect the normal behavior of Cygnus as the control port is only used to fetch the version, nothing important.
Have you tried sending some update context to Orion to get notified from Cygnus? Even by simulating a notification? Please see the Cygnus Quick Start Guide for an example on how to make such a simulation.
EDIT 2
Cygnus package names have es.tid.fiware.fiwareconnectors.cygnus...
been changed com.telefonica.iot.cygnus...
from version 0.8.0 (or possibly 0.9.0).
+1
source to share