<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	xmlns:series="https://publishpress.com/"
	>

<channel>
	<title>Migration Archives - 아빠프로그래머의 좌충우돌 개발하기!</title>
	<atom:link href="https://www.daddyprogrammer.org/post/category/migration/feed/" rel="self" type="application/rss+xml" />
	<link>https://www.daddyprogrammer.org/post/category/migration/</link>
	<description>기저귀 갈면서 코딩해야 프로다!</description>
	<lastBuildDate>Thu, 17 Feb 2022 09:52:25 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://www.daddyprogrammer.org/wp-content/uploads/2019/04/cropped-spring-bird-2295431_960_720-300x300.jpg</url>
	<title>Migration Archives - 아빠프로그래머의 좌충우돌 개발하기!</title>
	<link>https://www.daddyprogrammer.org/post/category/migration/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>AWS Database Migration Service &#8211; Migration auroradb to kafka</title>
		<link>https://www.daddyprogrammer.org/post/14637/aws-dms-auroradb-kafka-migration/</link>
					<comments>https://www.daddyprogrammer.org/post/14637/aws-dms-auroradb-kafka-migration/#comments</comments>
		
		<dc:creator><![CDATA[happydaddy]]></dc:creator>
		<pubDate>Thu, 28 Jan 2021 14:58:13 +0000</pubDate>
				<category><![CDATA[AWS]]></category>
		<category><![CDATA[Migration]]></category>
		<category><![CDATA[AuroraDB]]></category>
		<category><![CDATA[kafka]]></category>
		<category><![CDATA[DMS]]></category>
		<guid isPermaLink="false">http://3.36.120.4/?p=14637</guid>

					<description><![CDATA[<p>DMS AWS 서비스 중엔 DMS라는 재미있는 서비스가 있습니다. DMS는 AWS Database Migration Service(AWS DMS)의 약자로서 소스 데이터 스토어에서 대상 데이터 스토어로 마이그레이션하는 데 사용할 수 있는 웹 서비스입니다. 서비스를 운영하다보면 특정한 DB에서 다른 DB나 이종의 스토어로 데이터를 이관해야 하는 경우가 종종 발생 합니다. 자체 솔루션을 가지고 있지 않은경우 프로그래머나 DBA가 수작업으로 노가다를 해야하고 마이그레이션 이후 [&#8230;]</p>
<p>The post <a href="https://www.daddyprogrammer.org/post/14637/aws-dms-auroradb-kafka-migration/">AWS Database Migration Service &#8211; Migration auroradb to kafka</a> appeared first on <a href="https://www.daddyprogrammer.org">아빠프로그래머의 좌충우돌 개발하기!</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<h2 class="wp-block-heading">DMS</h2>



<p class="wp-block-paragraph">AWS 서비스 중엔 DMS라는 재미있는 서비스가 있습니다. DMS는 AWS Database Migration Service(AWS DMS)의 약자로서 소스 데이터 스토어에서 대상 데이터 스토어로 마이그레이션하는 데 사용할 수 있는 웹 서비스입니다. </p>



<figure class="wp-block-image size-large"><img fetchpriority="high" decoding="async" width="1350" height="450" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/product-page-diagram-AWS-DMS_database-consolidation.86418ac11df3b896a07ff4efad07854003644f3d.png" alt="" class="wp-image-14661" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/product-page-diagram-AWS-DMS_database-consolidation.86418ac11df3b896a07ff4efad07854003644f3d.png 1350w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/product-page-diagram-AWS-DMS_database-consolidation.86418ac11df3b896a07ff4efad07854003644f3d-1024x341.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/product-page-diagram-AWS-DMS_database-consolidation.86418ac11df3b896a07ff4efad07854003644f3d-768x256.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/product-page-diagram-AWS-DMS_database-consolidation.86418ac11df3b896a07ff4efad07854003644f3d-150x50.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/product-page-diagram-AWS-DMS_database-consolidation.86418ac11df3b896a07ff4efad07854003644f3d-300x100.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/product-page-diagram-AWS-DMS_database-consolidation.86418ac11df3b896a07ff4efad07854003644f3d-696x232.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/product-page-diagram-AWS-DMS_database-consolidation.86418ac11df3b896a07ff4efad07854003644f3d-1068x356.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/product-page-diagram-AWS-DMS_database-consolidation.86418ac11df3b896a07ff4efad07854003644f3d-1260x420.png 1260w" sizes="(max-width: 1350px) 100vw, 1350px" /></figure>



<p class="wp-block-paragraph">서비스를 운영하다보면 특정한 DB에서 다른 DB나 이종의 스토어로 데이터를 이관해야 하는 경우가 종종 발생 합니다. 자체 솔루션을 가지고 있지 않은경우 프로그래머나 DBA가 수작업으로 노가다를 해야하고 마이그레이션 이후 검증도 해야 하는 대단히 고된 작업입니다. 대용량 마이그레이션이 필요할 경우 서비스를 몇시간 이상 중단하고 작업해야 하는 것은 덤입니다.</p>



<p class="wp-block-paragraph">이러한 상황에서 Solution이 될 수 있는것이 DMS입니다. AWS DMS는 마이그레이션 프로젝트와 관련된 어렵거나 지루한 작업 중 많은 부분을 맡아 처리합니다. 또한 대상 스토어에 대한 깊은 지식이 없는 개발자도 손쉽게 데이터 이관을 할 수 있습니다. 놀라운 점은 DB to DB 뿐만 아니라 상당히 다양한 서비스로 데이터를 이관할 수 있도록 기능을 제공한다는 점 입니다. Source Endpoint로 사용 가능한 서비스와 Target Endpoint로 사용가능한 서비스는 다음과 같습니다.</p>



<h3 class="wp-block-heading">Source Endpoint 호환 데이터 베이스</h3>



<p class="wp-block-paragraph"><a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Sources.html">https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Sources.html</a></p>



<ul class="wp-block-list"><li>Amazon Aurora(MySQL 호환 데이터 원본으로 지원됨) MySQL</li><li>Amazon Aurora(PostgreSQL 호환 데이터 원본으로 지원됨) PostgreSQL</li><li>Amazon DocumentDB(MongoDB와 호환)</li><li>Amazon S3</li><li>Linux, UNIX 및 Windows용 IBM Db2 LUW</li><li>MariaDB(MySQL 호환 데이터 원본으로 지원됨) 버전 10.0.24 ~ 10.0.28, 10.1, 10.2 및 10.3, 10.3.13, 10.4.</li><li>Microsoft Azure SQL 데이터베이스.</li><li>Microsoft SQL Server</li><li>MongoDB 버전 3.x 및 4.0.</li><li>MySQL</li><li>Oracle</li><li>PostgreSQL</li><li>SAP Sybase ASE</li></ul>



<h3 class="wp-block-heading">Target Endpoint  호환 데이터 베이스</h3>



<p class="wp-block-paragraph"><a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Targets.html">https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Targets.html</a></p>



<ul class="wp-block-list"><li>Amazon Aurora(MySQL 호환 데이터 원본으로 지원됨) MySQL</li><li>Amazon Aurora MySQL Serverless</li><li>Amazon Aurora(PostgreSQL 호환 데이터 원본으로 지원됨) PostgreSQL</li><li>Amazon Aurora PostgreSQL Serverless</li><li>Amazon DocumentDB(MongoDB와 호환)</li><li>Amazon DynamoDB</li><li>Amazon Kinesis</li><li>Amazon Neptune</li><li>Amazon Redshift</li><li>Amazon S3</li><li>Elasticsearch Service</li><li>Kafka</li><li>MariaDB</li><li>Microsoft SQL Server</li><li>MySQL</li><li>Oracle</li><li>PostgreSQL</li><li>SAP Sybase ASE</li></ul>



<h2 class="wp-block-heading">Migration을 위한 단계별 작업</h2>



<p class="wp-block-paragraph">DMS는 다음과 같은 3가지 요소를 설정함으로서 마이그레이션 작업을 수행할 수 있습니다.</p>



<h3 class="wp-block-heading">Replication Instance (복제 인스턴스 생성)</h3>



<figure class="wp-block-image size-large is-resized"><img decoding="async" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-instance.png" alt="" class="wp-image-14660" width="495" height="143" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-instance.png 495w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-instance-150x43.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-instance-300x87.png 300w" sizes="(max-width: 495px) 100vw, 495px" /></figure>



<p class="wp-block-paragraph">Source Database에서 Target Database로 이관 하는 작업을 수행하는 서버 인스턴스 입니다. 마이그레이션 성능에 따라 여러가지 서버 사양을 선택하여 생성할 수 있습니다. </p>



<h3 class="wp-block-heading">Endpoint (마이그레이션 대상 설정)</h3>



<p class="wp-block-paragraph">Source 및 Target Database에 접근하기 위한 Endpoint 정보입니다. (Source/Destination Database의 host 및 id/password와 같은 접속 정보) 생성한 Endpoint는 Replication Instance에서 연결 가능한지 테스트 할 수 있습니다.</p>



<h3 class="wp-block-heading">Migration Tasks (마이그레이션 작업 설정)</h3>



<p class="wp-block-paragraph">위에서 생성한 Replication Instance와 Source/Target Endpoint정보를 이용하여 마이그레이션 작업을 만들고 수행합니다. Migration Task를 사용하면 마이그레이션할 스키마와 마이그레이션 유형을 지정할 수 있습니다. 기존 데이터를 마이그레이션하거나, 지속적 변경 사항을 복제하거나, 또는 데이터 변경 사항만 복제할 수 있습니다.</p>



<h2 class="wp-block-heading">Mysql Database -&gt; Kafka Topic Migration 실습</h2>



<p class="wp-block-paragraph">DMS를 사용하면 소스와 대상의 데이터 베이스 엔진이 다른 이기종 간의 데이터 마이그레이션이 가능합니다. 실습에서는 Aurora Mysql Database에서 MSK(Managed Service Apache Kafka)로의 데이터 마이그레이션을 진행해 보겠습니다.</p>



<p class="wp-block-paragraph">이미 Database와 Kafka가 준비된 상황이라면 Step1,2는 생략해도 됩니다.</p>



<h3 class="wp-block-heading">Step1. Aurora Mysql DataBase 생성</h3>



<p class="wp-block-paragraph"><span style="background-color:#ffbfb5" class="td_text_highlight_marker"><strong><span class="has-inline-color has-vivid-red-color">Mysql Database에서 DMS를 사용하여 마이그레이션을 하려면 binary logging(binlog)이 활성화 되어있어야 합니다.</span></strong></span> parameter_group을 하나 생성하여 Database 생성시 관련 설정을 적용해 줍니다.</p>



<p class="wp-block-paragraph">AWS Service &#8211; RDS &#8211; Parameter groups &#8211; Create parameter group을 클릭하여 새로운 parameter group을 생성합니다. 이때 Type은 DB Cluster Parameter Group을 선택해야 합니다. Group name과 Description은 적당한 내용을 입력하여 생성합니다.</p>



<figure class="wp-block-image size-large"><img decoding="async" width="1642" height="1110" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group.png" alt="" class="wp-image-14667" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group.png 1642w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-1024x692.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-768x519.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-1536x1038.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-150x101.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-300x203.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-696x470.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-1068x722.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-621x420.png 621w" sizes="(max-width: 1642px) 100vw, 1642px" /></figure>



<p class="wp-block-paragraph">생성한 parameter_group의 상세 정보로 들어가면 다음과 같은 화면이 나오는데 filter에 binlog_format이라고 입력합니다. 그러면 다음과 같이 1개의 항목이 나오는데 Edit Parameter를 클릭하여 Values를 ROW로 변경하고 저장합니다. 아래 그림은 설정 완료한 상태입니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="2048" height="655" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-2-2048x655.png" alt="" class="wp-image-14668" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-2-2048x655.png 2048w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-2-1024x328.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-2-768x246.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-2-1536x491.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-2-150x48.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-2-300x96.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-2-696x223.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-2-1068x342.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/parameter-group-2-1313x420.png 1313w" sizes="auto, (max-width: 2048px) 100vw, 2048px" /></figure>



<p class="wp-block-paragraph">Aurora Database를 생성합니다. AWS Service &#8211; RDS &#8211; Create database를 클릭하여 Aurora Mysql Database를 생성합니다. 실습에서는 대부분 기본 설정을 사용합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1564" height="1558" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-1.png" alt="" class="wp-image-14662" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-1.png 1564w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-1-1024x1020.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-1-300x299.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-1-768x765.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-1-1536x1530.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-1-150x149.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-1-696x693.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-1-1068x1064.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-1-422x420.png 422w" sizes="auto, (max-width: 1564px) 100vw, 1564px" /></figure>



<p class="wp-block-paragraph">Capacity type은 Provisioned를 선택합니다.<span style="background-color:#ffbfb5" class="td_text_highlight_marker"><strong><span class="has-inline-color has-vivid-red-color"> Serverless의 경우 아직은 Source 데이터베이스로 사용할 수 없습니다.</span></strong></span></p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1578" height="918" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-2.png" alt="" class="wp-image-14663" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-2.png 1578w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-2-1024x596.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-2-768x447.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-2-1536x894.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-2-150x87.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-2-300x175.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-2-696x405.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-2-1068x621.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-2-722x420.png 722w" sizes="auto, (max-width: 1578px) 100vw, 1578px" /></figure>



<p class="wp-block-paragraph">Templates 정보는 데이터베이스 이름과 접속할 계정 정보를 설정할 수 있습니다. 적당한 값을 입력합니다. </p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1556" height="1654" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-3.png" alt="" class="wp-image-14664" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-3.png 1556w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-3-963x1024.png 963w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-3-768x816.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-3-1445x1536.png 1445w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-3-150x159.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-3-300x319.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-3-696x740.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-3-1068x1135.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-3-395x420.png 395w" sizes="auto, (max-width: 1556px) 100vw, 1556px" /></figure>



<p class="wp-block-paragraph">DB instance size는 테스트이므로 제일 작은 사양을 선택합니다. Availability &amp; durability는 기본 설정을 사용합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1550" height="1056" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-4.png" alt="" class="wp-image-14665" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-4.png 1550w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-4-1024x698.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-4-768x523.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-4-1536x1046.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-4-150x102.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-4-218x150.png 218w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-4-300x204.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-4-696x474.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-4-1068x728.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-4-616x420.png 616w" sizes="auto, (max-width: 1550px) 100vw, 1550px" /></figure>



<p class="wp-block-paragraph">Connectivity 정보도 대부분 기본값을 사용합니다. 실습에서는 로컬 PC에서 접속할 수 있도록 Public access는 Yes로 설정합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1550" height="1562" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-5.png" alt="" class="wp-image-14666" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-5.png 1550w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-5-1016x1024.png 1016w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-5-768x774.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-5-1524x1536.png 1524w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-5-150x151.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-5-300x302.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-5-696x701.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-5-1068x1076.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-5-417x420.png 417w" sizes="auto, (max-width: 1550px) 100vw, 1550px" /></figure>



<p class="wp-block-paragraph">Additional configration을 클릭하여 항목을 엽니다. <strong><span style="background-color:#ffbfb5" class="td_text_highlight_marker"><span class="has-inline-color has-vivid-red-color">항목 중 DB Cluster parameter group을 위에서 설정한 Parameter group 설정으로 변경</span></span></strong>합니다. 나머지 값들은 기본 설정을 사용합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1548" height="790" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-6.png" alt="" class="wp-image-14669" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-6.png 1548w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-6-1024x523.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-6-768x392.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-6-1536x784.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-6-150x77.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-6-300x153.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-6-696x355.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-6-1068x545.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rds-create-6-823x420.png 823w" sizes="auto, (max-width: 1548px) 100vw, 1548px" /></figure>



<p class="wp-block-paragraph">Create database를 클릭하여 DB를 생성합니다. 생성하는데 시간이 걸리므로 바로 Step2를 진행합니다.</p>



<h3 class="wp-block-heading">Step2. MSK(Managed Service Kafka) 생성</h3>



<p class="wp-block-paragraph">AWS Service &#8211; MSK로 이동합니다. Kafka를 생성하기전 Cluster Configuration을 하나 생성합니다.</p>



<p class="wp-block-paragraph">Cluster configurations &#8211; Create cluster configuration을 클릭합니다. General 항목은 알맞게 입력하고 Configuration properties for revision 1의 code 항목에서 <span style="background-color:#ffbfb5" class="td_text_highlight_marker"><strong><span class="has-inline-color has-vivid-red-color">auto.create.topics.enable=true로 설정</span></strong></span>하고 Create를 클릭하여 설정을 저장합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1628" height="1496" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-configuration.png" alt="" class="wp-image-14670" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-configuration.png 1628w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-configuration-1024x941.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-configuration-768x706.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-configuration-1536x1411.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-configuration-150x138.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-configuration-300x276.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-configuration-696x640.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-configuration-1068x981.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-configuration-457x420.png 457w" sizes="auto, (max-width: 1628px) 100vw, 1628px" /></figure>



<p class="wp-block-paragraph">이제 MSK를 생성합니다. Clusters &#8211; Create Cluster를 클릭합니다.</p>



<p class="wp-block-paragraph">Create cluster with custom settings 를 선택합니다.</p>



<p class="wp-block-paragraph">General &#8211; Cluster name에 적당한 이름을 입력합니다.</p>



<p class="wp-block-paragraph">Configuration &#8211; Use a custom configuration을 선택 후 위에서 생성한 Configuration을 선택합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1626" height="756" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-2.png" alt="" class="wp-image-14674" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-2.png 1626w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-2-1024x476.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-2-768x357.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-2-1536x714.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-2-150x70.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-2-300x139.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-2-696x324.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-2-1068x497.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-2-903x420.png 903w" sizes="auto, (max-width: 1626px) 100vw, 1626px" /></figure>



<p class="wp-block-paragraph">실습에서는 기본 VPC를 사용합니다. Numbers of Zones는 2를 선택합니다. First Zone, Second Zone은 알맞게 선택합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1632" height="1444" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-3.png" alt="" class="wp-image-14675" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-3.png 1632w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-3-1024x906.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-3-768x680.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-3-1536x1359.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-3-150x133.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-3-300x265.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-3-696x616.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-3-1068x945.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-3-475x420.png 475w" sizes="auto, (max-width: 1632px) 100vw, 1632px" /></figure>



<p class="wp-block-paragraph">Broker type은 최소 사양인 kafka.t3.small을 선택합니다. Number of brokers per Zone은 1을 입력합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1626" height="1130" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-4.png" alt="" class="wp-image-14676" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-4.png 1626w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-4-1024x712.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-4-768x534.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-4-1536x1067.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-4-150x104.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-4-300x208.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-4-696x484.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-4-1068x742.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-4-604x420.png 604w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-4-100x70.png 100w" sizes="auto, (max-width: 1626px) 100vw, 1626px" /></figure>



<p class="wp-block-paragraph">EBS storage volume per broker는 1GiB를 입력합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1628" height="648" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-5.png" alt="" class="wp-image-14677" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-5.png 1628w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-5-1024x408.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-5-768x306.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-5-1536x611.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-5-150x60.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-5-300x119.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-5-696x277.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-5-1068x425.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-5-1055x420.png 1055w" sizes="auto, (max-width: 1628px) 100vw, 1628px" /></figure>



<p class="wp-block-paragraph">Encryption &#8211; Within the cluster &#8211; Enable encryption within the cluster는 체크 해제합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1626" height="1100" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-6.png" alt="" class="wp-image-14678" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-6.png 1626w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-6-1024x693.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-6-768x520.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-6-1536x1039.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-6-150x101.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-6-300x203.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-6-696x471.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-6-1068x723.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/kafka-create-6-621x420.png 621w" sizes="auto, (max-width: 1626px) 100vw, 1626px" /></figure>



<p class="wp-block-paragraph">나머지 설정은 기본값을 사용하고 Create cluster를 클릭하여 생성합니다. msk 생성에도 시간이 오래 걸리므로 Step3를 진행합니다.</p>



<h3 class="wp-block-heading">Step3. Replication Instance 생성</h3>



<p class="wp-block-paragraph">AWS Service &#8211; Database Migration Service &#8211; Replication instances 로 이동합니다.</p>



<p class="wp-block-paragraph">Create replication instance를 클릭합니다.</p>



<p class="wp-block-paragraph">Name에는 적당한 이름을 입력하고 Instance class는 실습이므로 최소 사양을 선택합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1632" height="1076" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-1.png" alt="" class="wp-image-14681" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-1.png 1632w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-1-1024x675.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-1-768x506.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-1-1536x1013.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-1-150x99.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-1-300x198.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-1-696x459.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-1-1068x704.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-1-637x420.png 637w" sizes="auto, (max-width: 1632px) 100vw, 1632px" /></figure>



<p class="wp-block-paragraph">VPC 는 default를 선택합니다. </p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1628" height="618" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-2.png" alt="" class="wp-image-14682" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-2.png 1628w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-2-1024x389.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-2-768x292.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-2-1536x583.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-2-150x57.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-2-300x114.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-2-696x264.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-2-1068x405.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/replica-2-1106x420.png 1106w" sizes="auto, (max-width: 1628px) 100vw, 1628px" /></figure>



<p class="wp-block-paragraph">나머지 설정은 기본값을 사용합니다. Create를 클릭하여 생성합니다. </p>



<h3 class="wp-block-heading">Step4. Source/Target Endpoint 생성</h3>



<p class="wp-block-paragraph">이 단계부터는 Aurora Database와 MSK(kafka)가 생성 완료 된 상태에서 진행 가능합니다. 먼저 Source endpoint를 생성합니다.</p>



<h4 class="wp-block-heading">Source Endpoint 생성</h4>



<p class="wp-block-paragraph">Endpoint &#8211; Create Endpoint를 클릭합니다. Select RDS DB instance를 선택하고 아래 셀렉트 박스에서 위에서 생성한 DB를 선택합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1634" height="780" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-1.png" alt="" class="wp-image-14683" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-1.png 1634w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-1-1024x489.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-1-768x367.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-1-1536x733.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-1-150x72.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-1-300x143.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-1-696x332.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-1-1068x510.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-1-880x420.png 880w" sizes="auto, (max-width: 1634px) 100vw, 1634px" /></figure>



<p class="wp-block-paragraph">Database 접속을 위한 정보를 입력합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1624" height="1520" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-2.png" alt="" class="wp-image-14684" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-2.png 1624w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-2-1024x958.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-2-768x719.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-2-1536x1438.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-2-150x140.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-2-300x281.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-2-696x651.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-2-1068x1000.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-2-449x420.png 449w" sizes="auto, (max-width: 1624px) 100vw, 1624px" /></figure>



<p class="wp-block-paragraph">Test endpoint connection(optional)를 클릭하여 펼칩니다. VPC는 기본을 선택하고 Replication Instance는 위에서 생성한 Instance를 선택하고 Run test를 클릭합니다. status 상태가 success 인것을 확인한다음 Create를 클릭하여 Endpoint를 생성합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1306" height="174" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-3.png" alt="" class="wp-image-14685" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-3.png 1306w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-3-1024x136.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-3-768x102.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-3-150x20.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-3-300x40.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-3-696x93.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/source-3-1068x142.png 1068w" sizes="auto, (max-width: 1306px) 100vw, 1306px" /></figure>



<h4 class="wp-block-heading">Target Endpoint 생성</h4>



<p class="wp-block-paragraph">Endpoint &#8211; Create Endpoint를 클릭합니다. 이번엔 Target endpoint를 선택합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1574" height="534" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-1.png" alt="" class="wp-image-14686" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-1.png 1574w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-1-1024x347.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-1-768x261.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-1-1536x521.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-1-150x51.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-1-300x102.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-1-696x236.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-1-1068x362.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-1-1238x420.png 1238w" sizes="auto, (max-width: 1574px) 100vw, 1574px" /></figure>



<p class="wp-block-paragraph">Endpoint identifier에는 적당한 이름을 입력합니다. Target engine은 Kafka를 선택합니다. Broker정보는 위에서 생성한 MSK의 broker정보를 입력합니다. Topic에는 마이그레이션 정보를 저장할 Topic이름을 입력합니다. 입력하지 않으면 시스템이 default(<strong><em>kafka</em>&#8211;<em>default</em>&#8211;<em>topic</em></strong>) topic을 생성합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1572" height="1258" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-2.png" alt="" class="wp-image-14687" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-2.png 1572w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-2-1024x819.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-2-768x615.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-2-1536x1229.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-2-150x120.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-2-300x240.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-2-696x557.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-2-1068x855.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-2-525x420.png 525w" sizes="auto, (max-width: 1572px) 100vw, 1572px" /></figure>



<p class="wp-block-paragraph">Test endpoint connection(optional)를 클릭하여 펼칩니다. VPC는 기본을 선택하고 Replication Instance는 위에서 생성한 Instance를 선택하고 Run test를 클릭합니다. status 상태가 success 인것을 확인한다음 Create를 클릭하여 Endpoint를 생성합니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1256" height="172" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-3.png" alt="" class="wp-image-14688" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-3.png 1256w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-3-1024x140.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-3-768x105.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-3-150x21.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-3-300x41.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-3-696x95.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/target-3-1068x146.png 1068w" sizes="auto, (max-width: 1256px) 100vw, 1256px" /></figure>



<h3 class="wp-block-heading">Step5. Migration Task 생성 및 마이그레이션 작업 진행</h3>



<p class="wp-block-paragraph">이제 Task만 작성하면 마이그레이션을 수행할 수 있습니다.</p>



<p class="wp-block-paragraph">Database migration tasks &#8211; Create task를 클릭합니다.</p>



<p class="wp-block-paragraph">task identifier에는 알맞은 값을 입력합니다. <br>Replication instance, Source database endpoint, Target database endpoint는 위에서 생성한 정보를 선택합니다.</p>



<p class="wp-block-paragraph">Mygration Type은 3가지가 있는데 알맞은 타입을 선택합니다.</p>



<ul class="wp-block-list"><li>Migrate existing data &#8211; 소스 엔드 포인트에서 대상 엔드 포인트로 일회성 마이그레이션을 수행합니다.</li><li>Migrate existing data and replicate ongoing changes &#8211; 소스에서 타겟으로 일회성 마이그레이션을 수행 한 다음 소스에서 타겟으로 데이터 변경 사항을 계속 복제합니다.</li><li>Replicate data changes only &#8211; 일회성 마이그레이션은 수행하지 않고 소스에서 대상으로 데이터 변경 사항만 계속 복제합니다.</li></ul>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1586" height="1228" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/task-1.png" alt="" class="wp-image-14689" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/task-1.png 1586w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/task-1-1024x793.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/task-1-768x595.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/task-1-1536x1189.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/task-1-150x116.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/task-1-300x232.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/task-1-696x539.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/task-1-1068x827.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/task-1-542x420.png 542w" sizes="auto, (max-width: 1586px) 100vw, 1586px" /></figure>



<h4 class="wp-block-heading">Table mappings 세팅</h4>



<p class="wp-block-paragraph">Selection rule 항목에서는 Source database에서 마이그레이션할 대상을 선택할 수 있습니다. 설정에 따라 데이터 베이스 전체 테이블을 복제할 수도 있고 특정 테이블만 복제할 수도 있습니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1688" height="1130" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-1.png" alt="" class="wp-image-14690" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-1.png 1688w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-1-1024x685.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-1-768x514.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-1-1536x1028.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-1-150x100.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-1-300x201.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-1-696x466.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-1-1068x715.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-1-627x420.png 627w" sizes="auto, (max-width: 1688px) 100vw, 1688px" /></figure>



<p class="wp-block-paragraph">Transformation rule에서는 마이그레이션 데이터에 대한 가공을 진행할 수 있습니다. 테이블 명을 변경한다던지 컬럼의 이름이나 데이터 타입을 변경할 수 있습니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1642" height="1336" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-2.png" alt="" class="wp-image-14691" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-2.png 1642w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-2-1024x833.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-2-768x625.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-2-1536x1250.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-2-150x122.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-2-300x244.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-2-696x566.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-2-1068x869.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/rule-2-516x420.png 516w" sizes="auto, (max-width: 1642px) 100vw, 1642px" /></figure>



<p class="wp-block-paragraph">Selection rule은 필수이므로 한가지 rule은 무조건 입력해야 합니다. 실습에서는 특정테이블 1개만 마이그레이션 하도록 rule을 생성하였습니다.<br>Create task를 클릭하여 task를 생성합니다. 생성이 완료되면 Actions &#8211; Restart/Resume을 선택하여 마이그레이션을 시작할 수 있습니다.</p>



<h3 class="wp-block-heading">Step6. 마이그레이션 데이터 확인</h3>



<p class="wp-block-paragraph">마이그레이션이 완료되면 다음과 같이 Status가 Replication~~ ongoing으로 변경됩니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="2048" height="303" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-1-2048x303.png" alt="" class="wp-image-14692" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-1-2048x303.png 2048w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-1-1024x152.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-1-768x114.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-1-1536x227.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-1-150x22.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-1-300x44.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-1-696x103.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-1-1068x158.png 1068w" sizes="auto, (max-width: 2048px) 100vw, 2048px" /></figure>



<p class="wp-block-paragraph">Table statistics 탭에서는 마이그레이션 된 테이블의 통계정보를 확인할 수 있습니다.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="2048" height="625" src="http://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-2-2048x625.png" alt="" class="wp-image-14693" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-2-2048x625.png 2048w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-2-1024x313.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-2-768x234.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-2-1536x469.png 1536w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-2-150x46.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-2-300x92.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-2-696x212.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-2-1068x326.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2021/01/mig-2-1376x420.png 1376w" sizes="auto, (max-width: 2048px) 100vw, 2048px" /></figure>



<p class="wp-block-paragraph">MSK(kafka) Topic에는 다음과 같은 포맷으로 메시지가 전달됩니다. data 항목에는 table의 정보가 json형태로 마이그레이션 되서 전달되고 metadata 항목에는  마이그레이션 데이터에 대한 기본 정보가 전달됩니다.</p>



<ul class="wp-block-list"><li>DDL : operation : drop-table, create-table</li><li>DML : operation : load, insert, update, delete</li></ul>



<pre class="EnlighterJSRAW" data-enlighter-language="json" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">// drop-table
{
	"metadata":	{
		"timestamp":	"2020-11-26T13:27:02.482358Z",
		"record-type":	"control",
		"operation":	"drop-table",
		"partition-key-type":	"task-id",
		"schema-name":	"kwtoon",
		"table-name":	"gift"
	}
}
// create-table
{
	"metadata":	{
		"timestamp":	"2020-11-26T13:27:02.489854Z",
		"record-type":	"control",
		"operation":	"create-table",
		"partition-key-type":	"task-id",
		"schema-name":	"kwtoon",
		"table-name":	"gift"
	}
}
// load
{
	"data":	{
		"id":	1,
		"name":	"coupon",
		"created":	"2020-11-26T12:38:28Z"
	},
	"metadata":	{
		"timestamp":	"2020-11-26T13:27:02.498033Z",
		"record-type":	"data",
		"operation":	"load",
		"partition-key-type":	"primary-key",
		"schema-name":	"kwtoon",
		"table-name":	"gift"
	}
}
// insert
{
	"data":	{
		"id":	2,
		"name":	"coupon",
		"created":	"2020-11-26T12:38:28Z"
	},
	"metadata":	{
		"timestamp":	"2020-12-08T05:30:59.118012Z",
		"record-type":	"data",
		"operation":	"insert",
		"partition-key-type":	"schema-table",
		"schema-name":	"kwtoon",
		"table-name":	"gift"
	}
}
// update
{
	"data":	{
		"id":	1,
		"name":	"cash",
		"created":	"2020-11-26T12:38:28Z"
	},
	"metadata":	{
		"timestamp":	"2020-11-26T13:29:24.751256Z",
		"record-type":	"data",
		"operation":	"update",
		"partition-key-type":	"schema-table",
		"schema-name":	"kwtoon",
		"table-name":	"gift",
		"transaction-id":	8589934811
	}
}
// delete
{
	"data":	{
		"id":	1,
		"name":	"cash",
		"created":	"2020-11-26T12:38:28Z"
	},
	"metadata":	{
		"timestamp":	"2020-11-26T13:30:15.674865Z",
		"record-type":	"data",
		"operation":	"delete",
		"partition-key-type":	"schema-table",
		"schema-name":	"kwtoon",
		"table-name":	"gift",
		"transaction-id":	8589935097
	}
}</pre>
<p>The post <a href="https://www.daddyprogrammer.org/post/14637/aws-dms-auroradb-kafka-migration/">AWS Database Migration Service &#8211; Migration auroradb to kafka</a> appeared first on <a href="https://www.daddyprogrammer.org">아빠프로그래머의 좌충우돌 개발하기!</a>.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://www.daddyprogrammer.org/post/14637/aws-dms-auroradb-kafka-migration/feed/</wfw:commentRss>
			<slash:comments>1</slash:comments>
		
		
		
		<series:name><![CDATA[Database Migration 실습]]></series:name>
	</item>
		<item>
		<title>Database Migration by Transactional Outbox Pattern</title>
		<link>https://www.daddyprogrammer.org/post/14068/database-migration-by-transactional-outbox/</link>
					<comments>https://www.daddyprogrammer.org/post/14068/database-migration-by-transactional-outbox/#comments</comments>
		
		<dc:creator><![CDATA[happydaddy]]></dc:creator>
		<pubDate>Sat, 05 Dec 2020 12:59:21 +0000</pubDate>
				<category><![CDATA[AWS]]></category>
		<category><![CDATA[Migration]]></category>
		<category><![CDATA[kafka]]></category>
		<category><![CDATA[Transactional outbox]]></category>
		<category><![CDATA[Lambda]]></category>
		<category><![CDATA[Node.js]]></category>
		<guid isPermaLink="false">http://3.36.120.4/?p=14068</guid>

					<description><![CDATA[<p>이전 실습까지는 데이터 마이그레이션을 위해 Database에서 제공하는 binlog나 DynamoDB/MongoDB에서 제공하는 Change Stream을 통해 변경 데이터를 처리할 수 있었습니다. 하지만 이렇게 시스템적으로 지원을 받지 못하는 경우는 어떻게 해야할까요. 여러가지 방법이 있겟지만 이번 장에서는 Transactional Outbox Pattern을 사용하는 방법에 대해 살펴보겠습니다. AWS의 Serverless AuroraDB는 인스턴스 및 용량 관리의 복잡성을 크게 줄여주는 장점이 있어 최근 많이 사용되는 추세입니다. [&#8230;]</p>
<p>The post <a href="https://www.daddyprogrammer.org/post/14068/database-migration-by-transactional-outbox/">Database Migration by Transactional Outbox Pattern</a> appeared first on <a href="https://www.daddyprogrammer.org">아빠프로그래머의 좌충우돌 개발하기!</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">이전 실습까지는 데이터 마이그레이션을 위해 Database에서 제공하는 binlog나 DynamoDB/MongoDB에서 제공하는 Change Stream을 통해 변경 데이터를 처리할 수 있었습니다. 하지만 이렇게 시스템적으로 지원을 받지 못하는 경우는 어떻게 해야할까요. 여러가지 방법이 있겟지만 이번 장에서는 Transactional Outbox Pattern을 사용하는 방법에 대해 살펴보겠습니다.</p>



<p class="wp-block-paragraph">AWS의 Serverless AuroraDB는 인스턴스 및 용량 관리의 복잡성을 크게 줄여주는 장점이 있어 최근 많이 사용되는 추세입니다. 그런데 Serverless AuroraDB의 경우 Classic AuroraDB에서 제공하는 binlog를 사용할 수가 없습니다. 즉 현재 시점에서는 AWS DMS(Database Migration Service)를 사용할 수 없어 이형의 플랫폼으로 Database를 마이그레이션 해야할 경우 어려움에 봉착하게 됩니다. 결국 시스템 측면에서 도움을 받을 수 없으면 소프트웨어 측면에서 문제를 해결할 수 밖에 없습니다. </p>



<h2 class="wp-block-heading">Transactional Outbox Pattern</h2>



<p class="wp-block-paragraph">Transactional Outbox는 특정 도메인 객체(테이블)의 변경사항이 발생할때 단일 트랜잭션내에서 동일 데이터베이스 내의 Outbox 테이블에 관련 정보를 저장하여 후속 이벤트를 처리하는 방법입니다. 해당 시나리오는 다음과 같습니다.</p>



<ul class="wp-block-list"><li>동일 Database내에 변경사항을 기록하는 Outbox Table을 생성</li><li><strong><span class="has-inline-color has-vivid-red-color">단일 트랜잭션 내에서 데이터 변경 + OutBox로 변경사항을 기록</span></strong>하는 액션을 수행. 동일 Database내에서 단일 트랙잭션으로 데이터를 처리하므로 변경 데이터를 일관성있게 처리 할 수 있음</li><li>Outbox 테이블에 기록된 변경 사항은 Message Relay를 통해 Message Broker로 발행</li><li>Message Broker에 발행된 내역은 Outbox 테이블에서 삭제</li><li>타 플랫폼에서는 Message Broker를 구독하여 변경 데이터를 전달받아 마이그레이션을 진행</li></ul>



<h3 class="wp-block-heading">Event Flow</h3>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1247" height="603" src="http://www.daddyprogrammer.org/wp-content/uploads/2020/12/transactional-outbox-flow.png" alt="" class="wp-image-14077" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2020/12/transactional-outbox-flow.png 1247w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/transactional-outbox-flow-1024x495.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/transactional-outbox-flow-768x371.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/transactional-outbox-flow-150x73.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/transactional-outbox-flow-300x145.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/transactional-outbox-flow-696x337.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/transactional-outbox-flow-1068x516.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/transactional-outbox-flow-869x420.png 869w" sizes="auto, (max-width: 1247px) 100vw, 1247px" /></figure>



<h3 class="wp-block-heading">Data Flow</h3>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1103" height="498" src="http://www.daddyprogrammer.org/wp-content/uploads/2020/12/TransactionalOutbox-kafka.png" alt="" class="wp-image-14078" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2020/12/TransactionalOutbox-kafka.png 1103w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/TransactionalOutbox-kafka-1024x462.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/TransactionalOutbox-kafka-768x347.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/TransactionalOutbox-kafka-150x68.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/TransactionalOutbox-kafka-300x135.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/TransactionalOutbox-kafka-696x314.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/TransactionalOutbox-kafka-1068x482.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2020/12/TransactionalOutbox-kafka-930x420.png 930w" sizes="auto, (max-width: 1103px) 100vw, 1103px" /></figure>



<h2 class="wp-block-heading">Create Outbox Table</h2>



<p class="wp-block-paragraph">마이그레이션 대상의 변경 이력을 저장하기 위한 Outbox 테이블을 생성합니다.</p>



<figure class="wp-block-table is-style-regular"><table><thead><tr><th>Field</th><th>Type</th><th>Comment</th><th>Sample</th></tr></thead><tbody><tr><td>id</td><td>bigint</td><td>PrimaryKey /AutoIncrement</td><td>1000</td></tr><tr><td>aggregate_id</td><td>varchar</td><td>순서 처리가 필요한 데이터를 Grouping하는 id( ex. user_id)</td><td>happydaddy</td></tr><tr><td>aggregate_type</td><td>varchar</td><td>변경이 발생한 도메인(테이블) 이름</td><td>LICENSE</td></tr><tr><td>payload</td><td>text</td><td>도메인 entity 변경사항(JSON)</td><td>{&#8220;id&#8221;:42,&#8221;user_id&#8221;:&#8221;happydaddy&#8221;,&#8221;episode_id&#8221;:14562,&#8221;expire_at&#8221;:&#8221;2021-01-01T15:23:38Z&#8221;}</td></tr><tr><td>event_type</td><td>varchar</td><td>발생한 이벤트 (INSERT, UPDATE, DELETE &#8230;)</td><td>INSERT</td></tr><tr><td>created_at_datetime</td><td>datetime</td><td>이벤트 발생시간</td><td>2020-12-01 23:38:39.616869</td></tr></tbody></table></figure>



<h2 class="wp-block-heading">Spring ApplicationEvent를 통한 Outbox 이벤트 처리</h2>



<p class="wp-block-paragraph">Java Spring 환경으로 작성되었으며 주요 부분에 대해서만 설명합니다. 전체 code는 다음 링크를 확인해 주십시오.</p>



<p class="wp-block-paragraph"><a href="https://github.com/codej99/TransactionalOutbox.git" target="_blank" rel="noreferrer noopener">https://github.com/codej99/TransactionalOutbox.git</a></p>



<h3 class="wp-block-heading">Application Event를 발행하는 event publisher 생성</h3>



<pre class="EnlighterJSRAW" data-enlighter-language="java" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">@Component
public class EventPublisher implements ApplicationEventPublisherAware {

    private ApplicationEventPublisher publisher;

    @Override
    public void setApplicationEventPublisher(ApplicationEventPublisher applicationEventPublisher) {
        this.publisher = applicationEventPublisher;
    }

    public void publish(OutBoxEvent outboxEvent) {
        this.publisher.publishEvent(outboxEvent);
    }
}</pre>



<h3 class="wp-block-heading">비즈니스 로직 처리시 테이블 변경사항이 발생하면 단일 트랜잭션 내에서 Outbox Event를 발행</h3>



<pre class="EnlighterJSRAW" data-enlighter-language="java" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">@RequiredArgsConstructor
@Service
public class LicenseService implements RegisterLicenseUseCase {

    private final RegisterLicensePort registerLicensePort;
    private final EventPublisher eventPublisher;

    @Override
    @Transactional
    public void registerLicense(License license) {
        LicenseEntity licenseEntity = LicenseEntity.builder().userid(license.getUserid()).episodeId(license.getEpisodeId()).expireAtDatetime(license.getExpireAtDatetime()).build();
        registerLicensePort.register(licenseEntity);
        eventPublisher.publish(EventUtils.createLicenseEvent(licenseEntity, EventUtils.EventType.INSERT));
    }
}</pre>



<h3 class="wp-block-heading">이벤트 리스너가 데이터베이스 변경 이벤트를 Outbox 테이블에 기록</h3>



<pre class="EnlighterJSRAW" data-enlighter-language="java" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">@RequiredArgsConstructor
@Service
public class EventService {

    private final SaveOutBoxPort saveOutBoxPort;

    @EventListener
    public void handleOutBoxEvent(OutBoxEvent event) {
        OutBoxEntity outBox = OutBoxEntity.builder()
                .aggregateId(event.getAggregateId())
                .aggregateType(event.getAggregateType().name())
                .eventType(event.getEventType().name())
                .payload(event.getPayload().toString())
                .createdAtDatetime(LocalDateTime.now())
                .build();
        saveOutBoxPort.save(outBox);
    }
}</pre>



<h2 class="wp-block-heading">Message Broker로 변경사항(Outbox)을 발행하는 Message Relay 작성</h2>



<p class="wp-block-paragraph">node로 작성되었으며 주요 부분에 대해서만 설명합니다. 전체 code는 다음 링크를 확인해 주십시오.</p>



<p class="wp-block-paragraph"><a href="https://github.com/codej99/TransactionalOutboxMessageRelay.git" target="_blank" rel="noreferrer noopener">https://github.com/codej99/TransactionalOutboxMessageRelay.git</a></p>



<h3 class="wp-block-heading">Outbox 테이블에 기록된 DB 변경 이벤트 조회</h3>



<pre class="EnlighterJSRAW" data-enlighter-language="js" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">const publishOutbox = () => {
    const connection = createConnection();
    const sql = `SELECT
                    id,
                    aggregate_id,
                    aggregate_type,
                    event_type,
                    payload,
                    created_at_datetime
                FROM outbox
                ORDER BY id asc`;
    connection.query(sql, function (err, result) {
        if (result) {
            pushToKafka(result);
        } else {
            console.error(`publishOutbox failed - ${err}`);
        }
    });
    connection.end(function (err) {
    });
};</pre>



<h3 class="wp-block-heading">kafka topic에 변경사항 발행</h3>



<pre class="EnlighterJSRAW" data-enlighter-language="js" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">const pushToKafka = (rows) => {
    const Producer = kafka.Producer,
        client = new kafka.KafkaClient({kafkaHost: process.env.KAFKA_BROKER}),
        producer = new Producer(client, {partitionerType: 3});

    let payloads = [];
    for (let i = 0; i &lt; rows.length; i++) {
        rows[i].payload = JSON.parse(rows[i].payload);
        console.log(rows[i].id + " : " + JSON.stringify(rows[i]));
        payloads.push({
            topic: process.env.KAFKA_TOPIC,
            messages: JSON.stringify(rows[i]),
            key: rows[i].aggregate_id
        })
    }
    producer.on("ready", function () {
        producer.send(payloads, function (err, result) {
            if (result) {
                console.info(`[SUCCESS] publishKafka - ${JSON.stringify(result)}`);
                deleteOutbox(rows);
            } else {
                console.error(`publishKafka failed - ${err}`);
            }
            producer.close();
        })
    });
    producer.on("error", function (err) {
        console.error(`Unknown Error Occured : ${err}`);
    });
};</pre>



<h3 class="wp-block-heading">발행한 이벤트를 Outbox에서 삭제</h3>



<pre class="EnlighterJSRAW" data-enlighter-language="js" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">const deleteOutbox = (rows) => {
    const connection = createConnection();
    for (let i = 0; i &lt; rows.length; i++) {
        const sql = `DELETE FROM outbox WHERE id=${rows[i].id}`;
        connection.query(sql, function (err, result) {
            if (result)
                console.info(`[SUCCESS] deleteOutbox - ${JSON.stringify(result)}`);
            else
                console.error(`deleteOutbox failed - ${err}`);
        });
    }
    connection.end(function (err) {
    });
};</pre>



<h2 class="wp-block-heading" id="참고">참고</h2>



<p class="wp-block-paragraph"><a href="https://dzone.com/articles/implementing-the-outbox-pattern" target="_blank" rel="noreferrer noopener">https://dzone.com/articles/implementing-the-outbox-pattern</a></p>



<p class="wp-block-paragraph"><a href="https://www.popit.kr/msa에서-메시징-트랜잭션-처리하기" target="_blank" rel="noreferrer noopener">https://www.popit.kr/msa에서-메시징-트랜잭션-처리하기</a></p>



<p class="wp-block-paragraph"><a href="https://medium.com/contino-engineering/publishing-events-to-kafka-using-a-outbox-pattern-867a48e29d35" target="_blank" rel="noreferrer noopener">https://medium.com/contino-engineering/publishing-events-to-kafka-using-a-outbox-pattern-867a48e29d35</a></p>
<p>The post <a href="https://www.daddyprogrammer.org/post/14068/database-migration-by-transactional-outbox/">Database Migration by Transactional Outbox Pattern</a> appeared first on <a href="https://www.daddyprogrammer.org">아빠프로그래머의 좌충우돌 개발하기!</a>.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://www.daddyprogrammer.org/post/14068/database-migration-by-transactional-outbox/feed/</wfw:commentRss>
			<slash:comments>1</slash:comments>
		
		
		
		<series:name><![CDATA[Database Migration 실습]]></series:name>
	</item>
		<item>
		<title>AuroraDB Migration &#8211; Using Kafka as a target for Database Migration Service</title>
		<link>https://www.daddyprogrammer.org/post/13969/aurora-db-migration-using-kafka/</link>
					<comments>https://www.daddyprogrammer.org/post/13969/aurora-db-migration-using-kafka/#comments</comments>
		
		<dc:creator><![CDATA[happydaddy]]></dc:creator>
		<pubDate>Sat, 28 Nov 2020 01:24:10 +0000</pubDate>
				<category><![CDATA[AWS]]></category>
		<category><![CDATA[Migration]]></category>
		<category><![CDATA[AuroraDB]]></category>
		<category><![CDATA[kafka]]></category>
		<guid isPermaLink="false">http://3.36.120.4/?p=13969</guid>

					<description><![CDATA[<p>AuroraDB(mysql)의 binary log(binlog)를 이용하면 database의 변경사항을 다른 시스템으로 쉽게 이관할 수 있습니다. binlog란 mysql에서 제공하는 데이터의 변경 event를 저장하는 로그 파일을 의미합니다. 데이터에 대한 변경 내역뿐 아니라 그 쿼리가 실행될 때 생성된 정보들도 가지고 있어 mysql의 서버 장애 시 복구를 위해 사용하거나 복제(replica) 데이터 구성을 위해 사용합니다. 본문에서는 AWS에서 제공하는 DMS(Database Migration Service) 기능을 이용하여 [&#8230;]</p>
<p>The post <a href="https://www.daddyprogrammer.org/post/13969/aurora-db-migration-using-kafka/">AuroraDB Migration &#8211; Using Kafka as a target for Database Migration Service</a> appeared first on <a href="https://www.daddyprogrammer.org">아빠프로그래머의 좌충우돌 개발하기!</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">AuroraDB(mysql)의 binary log(binlog)를 이용하면 database의 변경사항을 다른 시스템으로 쉽게 이관할 수 있습니다. binlog란 mysql에서 제공하는 데이터의 변경 event를 저장하는 로그 파일을 의미합니다. 데이터에 대한 변경 내역뿐 아니라 그 쿼리가 실행될 때 생성된 정보들도 가지고 있어 mysql의 서버 장애 시 복구를 위해 사용하거나 복제(replica) 데이터 구성을 위해 사용합니다.</p>



<p class="wp-block-paragraph">본문에서는 AWS에서 제공하는 DMS(Database Migration Service) 기능을 이용하여 AuroraDB의 binlog를 kafka로 이관해 볼 것입니다. Database의 복제본을 kafka로 이관하는 이유는 다양한 Cosumer가 스트림을 전달받아 용도에 맞게 데이터를 가공하기 위해서입니다. 예를 들면 통계 데이터 생성을 위한 원천 데이터가 필요할 수 있습니다. Database의 변경 스트림을 kafka로 전파하면 이형의 플랫폼(Hadoop과 같은 Bigdata Platform이나 Mongodb 같은 Nosql Platform)으로 데이터를 전파하여 용도에 맞게 데이터를 가공하여 사용할 수 있습니다.   </p>



<h2 class="wp-block-heading" id="Data-Flow">Data Flow</h2>



<p class="wp-block-paragraph">Aurora DB(Source Endpoint)의 변경사항을 DMS를 이용하여 Kafka(Destination Endpoint)에 발행합니다. 카프카의 Topic을 구독 중인 Consumer는 스트리밍 데이터를 전달받아 이형의 데이터 플랫폼에 전달하게 됩니다. </p>



<div class="wp-block-image"><figure class="aligncenter size-large is-resized"><img loading="lazy" decoding="async" src="http://www.daddyprogrammer.org/wp-content/uploads/2020/11/auroradb-dms-kafka.png" alt="" class="wp-image-13978" width="691" height="340" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2020/11/auroradb-dms-kafka.png 691w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/auroradb-dms-kafka-150x74.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/auroradb-dms-kafka-300x148.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/auroradb-dms-kafka-324x160.png 324w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/auroradb-dms-kafka-533x261.png 533w" sizes="auto, (max-width: 691px) 100vw, 691px" /></figure></div>



<h2 class="wp-block-heading">AuroraDB 생성시 체크사항</h2>



<p class="wp-block-paragraph">auroraDB를 migration 해야 하는 경우 binary logging이 enable 되어있어야 합니다. auroraDB의 경우 serverless 타입으로도 생성할 수 있는데 이 경우에는 binlog를 제공하지 않아 DMS로의 마이그레이션이 불가능하므로 serverless 방식으로 생성하면 안 됩니다. binary logging을 enable하는 방법은 db 클러스터에 적용된 parameter_group의 binlog_format 값을 ROW로 설정하면 됩니다. 기존의 db에 적용하는 경우는 재시작해야 설정이 적용됩니다.</p>



<div class="wp-block-image"><figure class="aligncenter size-large is-resized"><img loading="lazy" decoding="async" src="http://www.daddyprogrammer.org/wp-content/uploads/2020/11/rds-parameter-group.png" alt="" class="wp-image-14001" width="792" height="298" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2020/11/rds-parameter-group.png 792w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/rds-parameter-group-768x289.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/rds-parameter-group-150x56.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/rds-parameter-group-300x113.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/rds-parameter-group-696x262.png 696w" sizes="auto, (max-width: 792px) 100vw, 792px" /></figure></div>



<h2 class="wp-block-heading">MSK(Managed Service Kafka) 생성시 체크사항</h2>



<p class="wp-block-paragraph">MSK의 cluster configuration의 설정값 중 auto.create.topics.enable=true로 설정해야 합니다.</p>



<div class="wp-block-image"><figure class="aligncenter size-large"><img loading="lazy" decoding="async" width="555" height="341" src="http://www.daddyprogrammer.org/wp-content/uploads/2020/11/kafka-configuration.png" alt="" class="wp-image-14002" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2020/11/kafka-configuration.png 555w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/kafka-configuration-150x92.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/kafka-configuration-300x184.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/kafka-configuration-356x220.png 356w" sizes="auto, (max-width: 555px) 100vw, 555px" /></figure></div>



<p class="wp-block-paragraph">이관하려는 Database와 MSK의 VPC가 서로 다르다면 상태 변경 데이터를 이관할 수 없으므로 VPC Peering이나 Transit gateway를 적용해야 합니다.</p>



<h2 class="wp-block-heading">DMS 생성</h2>



<h3 class="wp-block-heading">Replication Instance 생성</h3>



<p class="wp-block-paragraph">auroraDB에서 binlog를 가져오거나 kafka의 topic에 binlog를 publish 할 인스턴스를 생성합니다. 설정 시 별다른 어려움은 없으며  적절한 Intance Type을 지정하고 VPC의 경우는 auroraDB와 MSK가 속해있는 vpc를 선택하면 됩니다.</p>



<h3 class="wp-block-heading">Endpoint 생성</h3>



<p class="wp-block-paragraph">위에서 생성한 replication instance를 이용하여 데이터 마이그레이션을 진행하게 되는데, 원천 데이터에 해당하는 Source Endpoint와 이관 대상이 되는 Target Endpoint 2개를 생성합니다.</p>



<h4 class="wp-block-heading">Source Endpoint 생성</h4>



<ul class="wp-block-list"><li>실습에서 Source Endpoint는 Aurora Mysql입니다.</li><li>Endpoint Type을 Source Endpoint로 체크하고 Select RDS DB Instance를 체크하면 이관하고자 하는 RDS를 선택할 수 있습니다. </li><li>Endpoint identifier는 적당한 레이블명으로 설정합니다.</li><li>Source engine은 AWS Aurora Mysql을 선택합니다.</li><li>Server name에는 DB의 Host명을 설정하고 접속 Port와 계정 정보(Name/Password)도 올바른 값으로 설정합니다. </li><li>그 외 값들은 기본값으로 세팅합니다.</li><li>Create Endpoint를 하기 전에 Test endpoint connection을 선택하여 replication instance에서 auroraDB로 접근이 가능한지 확인해야 합니다.</li><li>만약 접근이 불가능하다면 설정한 Host와 접속할 계정 정보가 맞는지 확인하고 DB의 Security Group의 Inbound rule에 3306 port가 열려있는지 확인합니다.</li></ul>



<h4 class="wp-block-heading">Target Endpoint 생성</h4>



<ul class="wp-block-list"><li>실습에서 Target Endpoint는 MSK입니다.</li><li>Endpoint Type을 Target Endpoint로 체크하고 Target Engine은 Kafka를 선택합니다.</li><li>Kafka Broker정보와 Topic 이름을 세팅합니다. Topic이름을 따로 지정하지 않으면 자동으로 생성하여 이관합니다. </li><li>그 외 값들은 기본값으로 세팅합니다.</li><li>Create Endpoint를 하기 전에 Test endpoint connection을 선택하여 replication instance에서 MSK로 접근이 가능한지 확인해야 합니다.</li><li>만약 접근이 불가능하다면 broker정보가 맞는지 확인하고 MSK의 Security Group의 Inbound rule에서 9092 port가 열려있는지 확인합니다. </li></ul>



<h3 class="wp-block-heading">Database migration tasks 생성</h3>



<p class="wp-block-paragraph">replication instance와 source, target endpoint를 모두 생성하였으면 드디어 Migration Task를 만들 수 있습니다.</p>



<ul class="wp-block-list"><li>replication instance에는 위에서 생성한 instance정보를 선택합니다.</li><li>source database endpoint에는 위에서 생성한 source endpoint를 선택합니다.</li><li>target database endpoint에는 위에서 생성한 target endpoint를 선택합니다.</li><li>migration type은 총 3가지인데 Migrate existing data and replicate ongoing changes를 선택합니다. 최초 한번 source 대상 전체를 migration하고 그 이후의 데이터 변경사항을 계속 복제합니다.</li><li>그 외의 설정은 기본값을 세팅합니다. 실제 서비스에서는 아래 링크의 내용을 참고하여 좀 더 세밀하게 설정하면 됩니다.</li></ul>



<p class="wp-block-paragraph"><a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html">Specifying task settings for AWS Database Migration </a><a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html" target="_blank" rel="noreferrer noopener">Service tasks &#8211; AWS Database Migration Service (amazon.com)</a></p>



<p class="wp-block-paragraph">migration task를 생성하면 즉시 Source -&gt; Target endpoint로 data migration이 수행됩니다. 만약 Staus가 fail일 경우엔 로그(Last failure message)를 확인하여 문제점을 수정하고 재시작하면 됩니다.</p>



<div class="wp-block-image"><figure class="aligncenter size-large is-resized"><img loading="lazy" decoding="async" src="http://www.daddyprogrammer.org/wp-content/uploads/2020/11/migration-task.png" alt="" class="wp-image-14005" width="1240" height="755" srcset="https://www.daddyprogrammer.org/wp-content/uploads/2020/11/migration-task.png 1240w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/migration-task-1024x623.png 1024w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/migration-task-768x468.png 768w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/migration-task-150x91.png 150w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/migration-task-300x183.png 300w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/migration-task-696x424.png 696w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/migration-task-1068x650.png 1068w, https://www.daddyprogrammer.org/wp-content/uploads/2020/11/migration-task-690x420.png 690w" sizes="auto, (max-width: 1240px) 100vw, 1240px" /></figure></div>



<h2 class="wp-block-heading" id="kafka-message-format">kafka message format</h2>



<p class="wp-block-paragraph">Migration task가 수행되면 원본 database에서 작업이 일어났을 때 DMS를 통해 kafka의 topic으로 변경사항이 발행(publish)됩니다. 발행된 데이터는 구독한 Consumer가 전달받을 수 있으며 메시지 형태는 다음과 같습니다.</p>



<ul class="wp-block-list"><li>DDL : operation &#8211; drop-table, create-table</li><li>DML : operation &#8211; load, update, delete</li></ul>



<pre class="EnlighterJSRAW" data-enlighter-language="json" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">{
	"metadata":	{
		"timestamp":	"2020-11-26T13:27:02.482358Z",
		"record-type":	"control",
		"operation":	"drop-table",
		"partition-key-type":	"task-id",
		"schema-name":	"kwtoon",
		"table-name":	"gift"
	}
}
{
	"metadata":	{
		"timestamp":	"2020-11-26T13:27:02.489854Z",
		"record-type":	"control",
		"operation":	"create-table",
		"partition-key-type":	"task-id",
		"schema-name":	"kwtoon",
		"table-name":	"gift"
	}
}
{
	"data":	{
		"id":	1,
		"name":	"coupon",
		"created":	"2020-11-26T12:38:28Z"
	},
	"metadata":	{
		"timestamp":	"2020-11-26T13:27:02.498033Z",
		"record-type":	"data",
		"operation":	"load",
		"partition-key-type":	"primary-key",
		"schema-name":	"kwtoon",
		"table-name":	"gift"
	}
}
{
	"data":	{
		"id":	1,
		"name":	"cash",
		"created":	"2020-11-26T12:38:28Z"
	},
	"metadata":	{
		"timestamp":	"2020-11-26T13:29:24.751256Z",
		"record-type":	"data",
		"operation":	"update",
		"partition-key-type":	"schema-table",
		"schema-name":	"kwtoon",
		"table-name":	"gift",
		"transaction-id":	8589934811
	}
}
{
	"data":	{
		"id":	1,
		"name":	"cash",
		"created":	"2020-11-26T12:38:28Z"
	},
	"metadata":	{
		"timestamp":	"2020-11-26T13:30:15.674865Z",
		"record-type":	"data",
		"operation":	"delete",
		"partition-key-type":	"schema-table",
		"schema-name":	"kwtoon",
		"table-name":	"gift",
		"transaction-id":	8589935097
	}
}</pre>



<h2 class="wp-block-heading" id="[참고]">[참고]</h2>



<p class="wp-block-paragraph"><a href="https://docs.aws.amazon.com/ko_kr/dms/latest/userguide/CHAP_Target.Kafka.html" target="_blank" rel="noreferrer noopener">https://docs.aws.amazon.com/ko_kr/dms/latest/userguide/CHAP_Target.Kafka.html</a></p>



<p class="wp-block-paragraph"><a href="https://github.com/aws-samples/aws-dms-msk-demo" target="_blank" rel="noreferrer noopener">https://github.com/aws-samples/aws-dms-msk-demo &#8211; Connect to preview</a></p>



<p class="wp-block-paragraph"><a href="https://docs.aws.amazon.com/ko_kr/msk/latest/developerguide/client-access.html" target="_blank" rel="noreferrer noopener">https://docs.aws.amazon.com/ko_kr/msk/latest/developerguide/client-access.html</a></p>



<p class="wp-block-paragraph"><a href="https://cloud.hosting.kr/aws-dms-database-migration-service/" target="_blank" rel="noreferrer noopener">https://cloud.hosting.kr/aws-dms-database-migration-service/</a></p>
<p>The post <a href="https://www.daddyprogrammer.org/post/13969/aurora-db-migration-using-kafka/">AuroraDB Migration &#8211; Using Kafka as a target for Database Migration Service</a> appeared first on <a href="https://www.daddyprogrammer.org">아빠프로그래머의 좌충우돌 개발하기!</a>.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://www.daddyprogrammer.org/post/13969/aurora-db-migration-using-kafka/feed/</wfw:commentRss>
			<slash:comments>1</slash:comments>
		
		
		
		<series:name><![CDATA[Database Migration 실습]]></series:name>
	</item>
	</channel>
</rss>
