Bourne's Blog - A Full-stack & Web3 Developer

「big data era」

Install Hadoop and related components automatically

Install Hadoop and related components automatically Install and configure hadoop is tedious, so I build a script to do this process automatically. The git project can be found here install hadoop...

Posted by Bourne's Blog - A Full-stack & Web3 Developer on May 2, 2022

HDFS commands

Most of the HDFS commands just like the same commands in linux, such as cat,chgrp,chmod,chown,cp,df,du,find,ls,mkdir,mv,rm,rmdir,stat, while also provide some other enhanced commands: Common comma...

Posted by Bourne's Blog - A Full-stack & Web3 Developer on May 2, 2022

Machine Learning - Pandas

1. Overview Pandas is a third-party Python library, together with Numpy and Matplotlib, is known as the three musketeers of data analysing. In the past, Python was mainly used for data collection ...

Posted by Bourne's Blog - A Full-stack & Web3 Developer on April 30, 2022

Kafka collect nginx log through flume

1. Create topic in Kafka 1 2 3 4 5 $ kafka-topics.sh --create \ --zookeeper hadoop001:2181 \ --replication-factor 2 \ --partitions 3 \ --topic gt_flume_data 2. Create flume configure file Speci...

Posted by Bourne's Blog - A Full-stack & Web3 Developer on April 25, 2022

Hive Windowing and Analytics Function - Part 1

Hive Windowing and Analytics Function - Part 1 Hive provides powerful window functions on current frame. In this chapter, I will demostrate how these functions work: first_value() last_value...

Posted by Bourne's Blog - A Full-stack & Web3 Developer on April 21, 2022

Create Hive UDF with IntelliJ

1. Create a maven project Don’t choose any template from the first dialog. Enter project name, such as “yukun-udf”; Add your identifier to group id like “org.yukun”. 2. Add dependency C...

Posted by Bourne's Blog - A Full-stack & Web3 Developer on April 21, 2022

Datawarehouse Model Design

Model design process seems like combine original tables with proper redundancy. For example, we have 3 tables in database: brands/corps/shops, Just put these fields together, and add table name a...

Posted by Bourne's Blog - A Full-stack & Web3 Developer on April 17, 2022

HIVE select output without header

Problem: Hive select output without column name 1 2 3 4 5 6 hive> select * from ods_crm_cdel_hold_da limit 10; OK 1 20001 7767 496 1 2021-11-23 11:25:27.0 2021-12-08 11:25:27.0 20220417 2 20002...

Posted by Bourne's Blog - A Full-stack & Web3 Developer on April 17, 2022

Max Compute - 搭建互联网在线运营分析平台

1. Aliyun Services involved Tablestore MaxCompute DataWorks QuickBI 2. Prepare data Create Tablestore instance and generate fake user data. step by step document is here. 3. Create tabl...

Posted by Bourne's Blog - A Full-stack & Web3 Developer on April 14, 2022

Hadoop Warnings

Hadoop Warnings After installing Hadoop and related coponents, when I run the hdfs command, I got warning: 1 2 3 4 [root@hadoop001 ~]# hdfs dfs -ls / Java HotSpot(TM) Server VM warning: You have l...

Posted by Bourne's Blog - A Full-stack & Web3 Developer on April 11, 2022