Onega

a lot of VC++ posts, a few C# posts, and some miscellaneous stuff

Sunday, April 27, 2014

Comparison of the Grid/Cloud Computing Frameworks

Krystian Lider: Comparison of the Grid/Cloud Computing Frameworks (Hadoop, GridGain, Hazelcast, DAC) - Part I
Comparison of the Grid/Cloud Computing Frameworks (Hadoop, GridGain, Hazelcast, DAC) - Part II

THE HADOOP DISTRIBUTED FILE SYSTEM
Konstantin Shvachko, Hairong Kuang,
Sanjay Radia, Robert Chansler
Presented by Alexander Pokluda
October 7, 2013

CBliag uDdatiaa Processing, 2013/14
Lecture 5: GFS & HDFS - distributed file systems!
Claudia Hauff (Web Information Systems)

PostgreSQL

Microsoft SQL Server vs. PostgreSQL
Oracle vs PostgreSQL Oracle vs. PostgreSQL

Cluster Computing and MapReduce bookmarks

Cluster Computing and MapReduce Lecture 1
    parallel vs distributed
Cluster Computing and MapReduce Lecture 2 
Mapper, Combiner, Reducer
Cluster Computing and MapReduce Lecture 3
NFS vs GFS
Cluster Computing and MapReduce Lecture 4
Clustering
Cluster Computing and MapReduce Lecture 5
Graph algorithm

More videos from Google developers.

Big Data Processing Lectures

StanfordUniversity
Introducing Apache Hadoop: The Modern Data Operating System

HADOOP
Introduction to HBase
Low latency, column based database.

Saturday, April 26, 2014

Ctrl+C handling

Handle SIGINT via ACE
extern "C" void handler (int)
{ACE_DEBUG ((LM_DEBUG, "%s\n", __FUNCTION__));
// shutdown application
}
ACE_Sig_Action sa ((ACE_SignalHandler) handler, SIGINT);

Handle SIGINT via boost.asio
void handler(
    const boost::system::error_code& error,
    int signal_number)
{
  if (!error)
  {
    // A signal occurred.
  }
}
boost::asio::signal_set signals(io_service, SIGINT, SIGTERM);
signals.async_wait(handler);

QT uses QSocketNotifier and sigaction together, see Calling Qt Functions From Unix Signal Handlers

Windows Console application, see CTRL_C_EVENT and SetConsoleCtrlHandler.

Sunday, April 20, 2014

TAO load balancing and fault tolerance

Keywords
GIOP::LOCATION_FORWARD PortableInterceptor::LOCATION_FORWARD
Examples
ACE / TAO / orbsvcs / LoadBalancer
ACE / TAO / examples / Load_Balancing
ACE / TAO / orbsvcs / examples / FaultTolerance
Paper
The Design and Performance of an Adaptive CORBA Load Balancing Service
Fault Tolerant (FT) CORBA Services
CORBA EXPLAINED SIMPLY by Ciaran McHale

Saturday, April 19, 2014

Use alternative gcc version

Sometimes gcc version of an OS is not up to date, especially when you are working on an OS of several years ago. Fortunately it is quite easy to build latest version on your own.
procedure to build gcc from source code is well documented on this page: http://gcc.gnu.org/wiki/InstallingGCC
tar xzf gcc-4.6.2.tar.gz
cd gcc-4.6.2
./contrib/download_prerequisites
cd ..
mkdir objdir
cd objdir
$PWD/../gcc-4.6.2/configure --prefix=$HOME/gcc-4.6.2
make
make install

I followed the above steps to build gcc-4.8.2.
[onega@workstation gcc482objdir]$  /home2/onega/src/gcc-4.8.2/configure --prefix=$HOME/gcc-4.8.2 && make && make install

I changed user-config.jam in order to use new gcc to build boost
[onega@workstation boost_1_55_0]$ grep "4.8.2" ./tools/build/v2/user-config.jam
using gcc : 4.8.2 : /home/onega/gcc-4.8.2/bin/g++ : -L/home/onega/gcc-4.8.2/lib -I/home/onega/gcc-4.8.2/include ;

/home2/onega/src/boost_1_55_0/bootstrap.sh --with-toolset=gcc --prefix=$HOME/boost_1_55_0
it will generate /home2/onega/src/boost_1_55_0/project-config.jam
(Some people suggested to modifying project-config.jam, but I didn't find it at first so I changed user-config.jam.)

./b2 --toolset=gcc-4.8.2 --with-date_time --with-thread --with-regex --with-system --with-filesystem --with-program_options install --prefix=$HOME/boost_1_55_0

To use gcc-4.8.2 in CDT
Open project properties window (Alt+Enter)
C/C++ Build->Settings->C++ Compiler->Command: ~/gcc-4.8.2/bin/g++
C/C++ Build->Settings->C++ Linker->Command: ~/gcc-4.8.2/bin/g++

Use different severity_level for different logger instance

Traditional log function supports log level, which is similar to boost::log::trivial::severity_level.
void simple_log(int log_level, char* fmt, ...);
The problem with traditional log function is that all modules are using the same log level, when debug level is enabled, there are too many log messages which may potentially change program behavior.
with boot log library it is easier to control log level per module.
BOOST_LOG_INLINE_GLOBAL_LOGGER_CTOR_ARGS(mylog1, my_logger_mt, (boost::log::keywords::channel = "mylog1"))
BOOST_LOG_INLINE_GLOBAL_LOGGER_CTOR_ARGS(mylog2, my_logger_mt, (boost::log::keywords::channel = "mylog2"))

module1 uses mylog1 to output message; module2 uses mylog2 to output message.
    BOOST_LOG_SEV(mylog2::get(), boost::log::trivial::warning) << "from mylog2 " << boost::posix_time::second_clock::local_time();
    BOOST_LOG_SEV(mylog1::get(), boost::log::trivial::info) << "from mylog1 " << boost::posix_time::second_clock::local_time();
Complete source code.
#include <cstddef>
#include <string>
#include <fstream>
#include <iomanip>
#include <boost/utility/empty_deleter.hpp>
#include <boost/log/core.hpp>
#include <boost/log/trivial.hpp>
#include <boost/log/sinks.hpp>
#include <boost/log/support/date_time.hpp>
#include <boost/log/common.hpp>
#include <boost/log/expressions.hpp>
#include <boost/log/attributes.hpp>
#include <boost/log/utility/setup/common_attributes.hpp>
#include <boost/log/utility/setup/formatter_parser.hpp>
#include <boost/log/sources/severity_logger.hpp>
#include <boost/log/sources/severity_feature.hpp>
#include <boost/log/sources/channel_feature.hpp>
#include <boost/log/sources/channel_logger.hpp>
#include <boost/log/sources/logger.hpp>
#include <boost/log/sources/record_ostream.hpp>
#include <boost/log/sources/global_logger_storage.hpp>
#include <boost/thread/thread.hpp>
#include <sys/types.h>
namespace logging = boost::log;
namespace sinks = boost::log::sinks;

typedef boost::log::sources::severity_channel_logger_mt<
        logging::trivial::severity_level,     // the type of the severity level
    std::string         // the type of the channel name
> my_logger_mt;

BOOST_LOG_INLINE_GLOBAL_LOGGER_CTOR_ARGS(mylog1, my_logger_mt, (boost::log::keywords::channel = "mylog1"))
BOOST_LOG_INLINE_GLOBAL_LOGGER_CTOR_ARGS(mylog2, my_logger_mt, (boost::log::keywords::channel = "mylog2"))

BOOST_LOG_ATTRIBUTE_KEYWORD(thread_id, "ThreadID", logging::attributes::current_thread_id::value_type)
void init_logging()
{
    boost::shared_ptr< logging::core > core = logging::core::get();
    boost::shared_ptr< sinks::text_ostream_backend > backend =
        boost::make_shared< sinks::text_ostream_backend >();
    backend->add_stream(boost::shared_ptr< std::ostream >(&std::clog, boost::empty_deleter()));
    backend->add_stream(boost::shared_ptr< std::ostream >(new std::ofstream("sample.log", std::ofstream::app)));
    backend->auto_flush(true);
    typedef sinks::synchronous_sink< sinks::text_ostream_backend > sink_t;
    boost::shared_ptr< sink_t > sink(new sink_t(backend));
    sink->set_formatter
        (
        boost::log::expressions::stream
        << boost::log::expressions::format_date_time< boost::posix_time::ptime >("TimeStamp", "%Y%m%d %H:%M:%S")
        << " "<< boost::log::expressions::attr<boost::log::attributes::current_thread_id::value_type>("ThreadID")
        << " "<< boost::log::expressions::attr< std::string >("Channel")<< " "
                       << ": <" << boost::log::trivial::severity
                       << "> " << boost::log::expressions::smessage
        );
    core->add_sink(sink);
core->set_filter
   (
    (boost::log::expressions::attr< std::string >("Channel") == "mylog1" &&
       logging::trivial::severity >= logging::trivial::info) ||
       (boost::log::expressions::attr< std::string >("Channel") == "mylog2" &&
               logging::trivial::severity >= logging::trivial::trace) ||
               logging::trivial::severity >= logging::trivial::error ||
               !boost::log::expressions::has_attr("Channel")
   );
boost::log::add_common_attributes();
logging::core::get()->add_global_attribute("ThreadID",logging::attributes::current_thread_id());

}

struct callable
{
callable()
{
BOOST_LOG_SEV(mylog1::get(), boost::log::trivial::info) << __FUNCTION__;
}
callable(const callable& aaa)
{
BOOST_LOG_SEV(mylog1::get(), boost::log::trivial::info) << __FUNCTION__ << " copy ctor.";
}
~callable()
{
BOOST_LOG_SEV(mylog1::get(), boost::log::trivial::info) << __FUNCTION__;
}
    void operator()()
    {
    BOOST_LOG_SEV(mylog1::get(), boost::log::trivial::info) << __FUNCTION__ << " thread id is " << std::hex<< (unsigned int)pthread_self();
    }
};

boost::thread copies_are_safe()
{
    callable x;
    return boost::thread(x);
}

int main(int argc, char* argv[]) {
init_logging();
std::cout << "Each logger use its own channel, check sample.log" << std::endl;
    BOOST_LOG_TRIVIAL(trace) << "A trace severity message";
    BOOST_LOG_TRIVIAL(debug) << "A debug severity message";
    BOOST_LOG_TRIVIAL(info) << argv[0] << " start at "
    << boost::posix_time::second_clock::local_time()
    << ", program was built at "<<__DATE__ <<" " <<__TIME__ <<" by gcc "<<__VERSION__;
    BOOST_LOG_TRIVIAL(error) << "An error severity message";
    BOOST_LOG_TRIVIAL(fatal) << "A fatal severity message";
    BOOST_LOG_SEV(mylog2::get(), boost::log::trivial::warning) << "from mylog2 " << boost::posix_time::second_clock::local_time();
    BOOST_LOG_SEV(mylog1::get(), boost::log::trivial::info) << "from mylog1 " << boost::posix_time::second_clock::local_time();
    BOOST_LOG_SEV(mylog1::get(), boost::log::trivial::trace) << "no trace from mylog1";
    boost::thread mythread=copies_are_safe();
    mythread.join();
return 0;
}
/*
[onega@workstation Debug]$ LD_LIBRARY_PATH=~/boost_1_55_0/lib:~/gcc-4.8.2/lib ./multilog
Each logger use its own channel, check sample.log
20140418 17:21:29 0xb7f016d0  : <trace> A trace severity message
20140418 17:21:29 0xb7f016d0  : <debug> A debug severity message
20140418 17:21:29 0xb7f016d0  : <info> ./multilog start at 2014-Apr-18 17:21:29, program was built at Apr 18 2014 17:21:19 by gcc 4.8.2
20140418 17:21:29 0xb7f016d0  : <error> An error severity message
20140418 17:21:29 0xb7f016d0  : <fatal> A fatal severity message
20140418 17:21:29 0xb7f016d0 mylog2 : <warning> from mylog2 2014-Apr-18 17:21:29
20140418 17:21:29 0xb7f016d0 mylog1 : <info> from mylog1 2014-Apr-18 17:21:29
20140418 17:21:29 0xb7f016d0 mylog1 : <info> callable
20140418 17:21:29 0xb7f016d0 mylog1 : <info> callable copy ctor.
20140418 17:21:29 0xb7f016d0 mylog1 : <info> callable copy ctor.
20140418 17:21:29 0xb7f016d0 mylog1 : <info> callable copy ctor.
20140418 17:21:29 0xb7f016d0 mylog1 : <info> callable copy ctor.
20140418 17:21:29 0xb7f016d0 mylog1 : <info> ~callable
20140418 17:21:29 0xb7f016d0 mylog1 : <info> ~callable
20140418 17:21:29 0xb7f016d0 mylog1 : <info> ~callable
20140418 17:21:29 0xb7f016d0 mylog1 : <info> ~callable
20140418 17:21:29 0xb7f00b90 mylog1 : <info> operator() thread id is b7f00b90
20140418 17:21:29 0xb7f016d0 mylog1 : <info> ~callable


Invoking: GCC C++ Compiler
~/gcc-4.8.2/bin/g++ -DBOOST_ALL_DYN_LINK -I/home/onega/boost_1_55_0/include -O0 -g3 -c -fmessage-length=0 -Wunused-variable -MMD -MP -MF"src/multilog.d" -MT"src/multilog.d" -o "src/multilog.o" "../src/multilog.cpp"
Invoking: GCC C++ Linker
~/gcc-4.8.2/bin/g++ -L/home/onega/boost_1_55_0/lib -o "multilog"  ./src/multilog.o   -lboost_log -lboost_thread -lboost_chrono -lboost_filesystem -lboost_date_time -lboost_system

Invoking: Cygwin C++ Linker
g++ -L"C:\Users\onega.zhang\workspace\boost_1_55_0\stage\lib" -o "multilog.exe"  ./src/multilog.o   -lboost_log-mt -lboost_thread-mt -lboost_date_time -lboost_filesystem-mt -lboost_system-mt
Invoking: Cygwin C++ Compiler
g++ -DBOOST_ALL_DYN_LINK -I"C:\Users\onega.zhang\workspace\boost_1_55_0" -O0 -g3 -c -fmessage-length=0 -Wunused-local-typedefs -Wunused-value -MMD -MP -MF"src/multilog.d" -MT"src/multilog.d" -o "src/multilog.o" "../src/multilog.cpp"

onega.zhang@R9REZAF /cygdrive/c/Users/onega.zhang/workspace/boost_1_55_0/stage/lib
$ /cygdrive/c/Users/onega.zhang/workspace/multilog/Debug/multilog.exe
Each logger use its own channel, check sample.log
A trace severity message
A debug severity message
/cygdrive/c/Users/onega.zhang/workspace/multilog/Debug/multilog start at 2014-Apr-17 16:01:30
A warning severity message
An error severity message
A fatal severity message
from mylog2
from mylog1 2014-Apr-17 16:01:30
*/