ããã«ã¡ã¯ãHabrïŒ ä»æ¥ãAmazon Web ServicesãæäŸããæ©äŒãšãã¢ããªã±ãŒã·ã§ã³ã®åé¡ã解決ããéã«ãããã®æ©äŒãã©ã®ããã«äœ¿çšãããã«ã€ããŠã®è°è«ãç¶ããŸãã
ç°¡åãªäŸã䜿çšããŠãæ°åã§ãã±ãŒã¹è§£æãåããç¬èªã®ãµãŒããŒã¬ã¹èªåã¹ã±ãŒã©ãã«REST APIãäœæããããšãæ€èšããŠãã ãã-ãªãœãŒã¹ã®ãªã¹ããååŸããŸãã
é¢çœãïŒ ãã®åŸãã«ããã®äžã«è¡ããŸãïŒ
ïŒåºå
žïŒåå ãã代ããã«
ãµã³ãã«ã解æããããã«ããŒã¿ããŒã¹ã䜿çšããã®ã§ã¯ãªããæ
å ±ã®ãœãŒã¹ã¯AWS S3ã®ãã¬ãŒã³ããã¹ããã¡ã€ã«ã«ãªããŸãã
- ãããã£ãŠãAWS S3ã«ã¯ããããŒä»ãã®ããã¹ããã¡ã€ã«ããããäœããã®ããã»ã¹ãããã«æ
å ±ãæžã蟌ããšããŸãã
- ã¯ã©ãŠãAPIãäœæããŸããããã¯ãæž¡ããããã©ã¡ãŒã¿ãŒã䜿çšããŠJSONå¿çãGETãªã¯ãšã¹ããšããŠè¿ããŸãã
- åæã«ãã¿ã¹ã¯ã®è€éããšãçµæãšããŠãªãœãŒã¹ã®åŠçèœåã«å¯ŸããèŠä»¶ã®å¢å ã«å¿ããŠãå¿é
ããå¿
èŠã¯ãããŸããã ãµãŒãã¹ã¯å®å
šã«èªåæ¡åŒµå¯èœã§ãã ããã¯ã管çããµãŒããŒã®å²ãåœãŠã管çãäžèŠã§ãã³ãŒããã¢ããããŒãããŠå®è¡ããã ãã§ããããšãæå³ããŸãã
éçºäžã®ã·ã¹ãã ã®ã¢ãŒããã¯ãã£

䜿çšãããã¢ããŸã³ãŠã§ããµãŒãã¹ã³ã³ããŒãã³ãïŒ
- Amazon S3-ãªããžã§ã¯ãã¹ãã¬ãŒãžãããã«ãããäºå®äžç¡å¶éã®éã®æ
å ±ãä¿åã§ããŸãã
- AWS Identity and Access ManagementïŒIAMïŒ -AWSã®ãµãŒãã¹ãšãªãœãŒã¹ãžã®ã¢ã¯ã»ã¹ãå®å
šã«å¶åŸ¡ããæ©èœãæäŸãããµãŒãã¹ã IAMã䜿çšããŠãAWSãŠãŒã¶ãŒãšã°ã«ãŒããäœæã管çã管çããã¢ã¯ã»ã¹èš±å¯ã䜿çšããŠAWSãªãœãŒã¹ãžã®ã¢ã¯ã»ã¹ãèš±å¯ãŸãã¯æåŠã§ããŸãã
- AWS Lambdaã¯ããµãŒããŒãããã¯ã¢ããããã³æ§æããã«ã³ãŒããå®è¡ã§ãããµãŒãã¹ã§ãã ãã¹ãŠã®èšç®èœåã¯ãã³ãŒã«ããšã«èªåçã«ã¹ã±ãŒãªã³ã°ãããŸãã æéã¯ãæ©èœãªã¯ãšã¹ãã®æ°ãšãã®æéã«åºã¥ããŠè«æ±ãããŸãã ã³ãŒããå®è¡ãããæéã
ç¡æã¢ã¯ã»ã¹ã®ã¬ãã«ïŒç¡æå©çšæ ïŒã¯ãç¡æã§40äžGb-sã®æããã100äžã®ãªã¯ãšã¹ããæå³ããŸãã ãµããŒããããŠããèšèªïŒNode.jsãJavaãCïŒãGoãPythonãRubyãPowerShell
ã Pythonã䜿çšããŸãã
- boto3ã©ã€ãã©ãªã¯ãããŸããŸãªAmazonãµãŒãã¹ãšããåãã§ããPythonçšAWS SDKã§ãã
- Amazon API Gatewayã¯ãããããèŠæš¡ã®APIãäœæãå
¬éãä¿å®ãç£èŠãããã³ä¿è·ããããã«èšèšãããå®å
šã«ç®¡çãããéçºè
ãµãŒãã¹ã§ãã ãããã°ãæ¹è¯ããã¹ãã®ããã«åãAPIïŒã¹ããŒãžïŒã®è€æ°ã®ããŒãžã§ã³ã䜿çšããæ©èœã«å ããŠããã®ãµãŒãã¹ã§ã¯AWS Lambdaã䜿çšããŠãµãŒããŒã¬ã¹REST APIãäœæã§ããŸãã Lambdaã¯ãéåžžã«ã¢ã¯ã»ã¹ããããã³ã³ãã¥ãŒãã£ã³ã°ã€ã³ãã©ã¹ãã©ã¯ãã£ã§ã³ãŒããå®è¡ããããããµãŒããŒã®é
åžãã¹ã±ãŒãªã³ã°ã管çã®å¿
èŠããªããªããŸãã
HTTP / REST APIã®ç¡æå©çšæ ã«ã¯ã12ãæéã1ãæããã100äžåã®APIåŒã³åºããå«ãŸããŸã
ããŒã¿æºå
ã¿ãä»ãã®ããã¹ããã¡ã€ã«ã¯ãGET RESTèŠæ±ãžã®å¿çãçæããããã®æ
å ±ã®ãœãŒã¹ãšããŠããã£ãŒã«ãåºåãæåãšããŠäœ¿çšãããŸãã çŸåšããã®äŸã§ã¯æ
å ±ã¯ããã»ã©éèŠã§ã¯ãããŸããããAPIãããã«äœ¿çšããããã«ãQuikãã¬ãŒãã£ã³ã°ã¿ãŒããã«ãããã·ã¢ã«ãŒãã«å»ºãŠã®åµåžã®çŸåšã®ååŒã®ããŒãã«ãããŠã³ããŒãããbonds.txtã«ä¿åãããã®ãã¡ã€ã«ãç¹å¥ã«äœæããAWS S3ãã±ããã«é
眮ããŸããã
åä¿¡ããæ
å ±ã®äŸã¯ã次ã®å³ã«ç€ºããšããã§ãã

次ã«ãbonds.txtãã¡ã€ã«ããæ
å ±ãèªã¿åãã解æããŠèŠæ±ã«å¿ããŠçºè¡ããé¢æ°ãäœæããå¿
èŠããããŸãã AWS Lambdaã¯ããã§åé¡ãªãåäœããŸãã ãã ããæåã«ãäœæããLambdaé¢æ°ãAWS S3ã«ãããã±ããããæ
å ±ãèªã¿åããããã«ããæ°ããããŒã«ãäœæããå¿
èŠããããŸãã
AWS Lambdaã®ããŒã«ãäœæãã
- AWS管çã³ã³ãœãŒã«ã§ãAWS IAMãµãŒãã¹ã«ç§»åãã[ããŒã«]ã¿ãã«ç§»åããŠã[ããŒã«ã®äœæ]ãã¿ã³ãã¯ãªãã¯ããŸãã
æ°ãã圹å²ãè¿œå ãã - çŸåšäœæããŠããããŒã«ã¯ãAWS Lambdaã«ãã£ãŠAWS S3ããæ
å ±ãèªã¿åãããã«äœ¿çšãããŸãã ãããã£ãŠã次ã®ã¹ãããã§ã¯ããä¿¡é Œãããã¿ã€ãã®éžæã->ãAWSãµãŒãã¹ãããã³ããã®ããŒã«ã䜿çšãããµãŒãã¹ã®éžæã->ãLambdaããéžæããã次ãžïŒèš±å¯ããã¿ã³ãã¯ãªãã¯ããŸã
LambdaãµãŒãã¹ã®åœ¹å² - 次ã«ãæ°ããäœæãããããŒã«ã§äœ¿çšãããAWSãªãœãŒã¹ã®ã¢ã¯ã»ã¹ããªã·ãŒãèšå®ããå¿
èŠããããŸãã ãªããªã ããªã·ãŒã®ãªã¹ãã¯éåžžã«å°è±¡çã§ãããªã·ãŒã®ãã£ã«ã¿ãŒã䜿çšããŠããS3ãã瀺ããŠããŸãã ãã®çµæãS3ãµãŒãã¹ã«é¢é£ããŠãã£ã«ã¿ãŒåŠçããããªã¹ããååŸããŸãã ãAmazonS3ReadOnlyAccessãããªã·ãŒã®å察åŽã®ãã§ãã¯ããã¯ã¹ã«æ³šæããã次ãžïŒã¿ã°ããã¿ã³ãã¯ãªãã¯ããŸãã
- ã¹ãããïŒã¿ã°ã®è¿œå ïŒãªãã·ã§ã³ïŒïŒã¯ãªãã·ã§ã³ã§ãããå¿
èŠã«å¿ããŠããŒã«ã®ã¿ã°ãæå®ã§ããŸãã ããã¯è¡ããã次ã®ã¹ãããã§ãããã¬ãã¥ãŒã«é²ã¿ãŸãã ããã§ãããŒã«åãèšå®ããå¿
èŠããããŸã-ãForLambdaS3-ReadOnlyãã説æãè¿œå ãããããŒã«ã®äœæããã¿ã³ãã¯ãªãã¯ããŸãã
ãã¹ãŠã圹å²ãäœæãããŠãããä»åŸã®äœæ¥ã§äœ¿çšã§ããŸãã
AWS Lambdaã§æ°ããæ©èœãäœæãã
- AWS LambdaãµãŒãã¹ã«ç§»åãã[é¢æ°ã®äœæ]ãã¿ã³ãã¯ãªãã¯ããŸãã
以äžã®ã¹ã¯ãªãŒã³ã·ã§ããã«ç€ºãããã«ããã¹ãŠã®ãã£ãŒã«ãã«å
¥åããŸãã
- åå-"getAllBondsList";
- ã©ã³ã¿ã€ã -ãPython 3.6ã
- 圹å²-ãæ¢åã®åœ¹å²ãéžæããã
- æ¢åã®ããŒã«-ããã§ãäžã§äœæããããŒã«ãéžæããŸã-ForLambdaS3-ReadOnly
- é¢æ°ã³ãŒããèšè¿°ããããŸããŸãªãã¹ãèµ·åã§ãã®ããã©ãŒãã³ã¹ããã§ãã¯ããã ãã§ãã Lambdaé¢æ°ã®äž»èŠã³ã³ããŒãã³ãïŒPythonã䜿çšããå ŽåïŒã¯boto3ã©ã€ãã©ãªã§ããããšã«æ³šæããŠãã ããã
import boto3 s3 = boto3.resource('s3') bucket = s3.Bucket('your-s3-bucket') obj = bucket.Object(key = 'bonds.txt') response = obj.get()
Pythoné¢æ°ã®åºæ¬çãªèãæ¹ã¯æ¬¡ã®ãšããã§ãã
- Bonds.txtãã¡ã€ã«ãéããŸãã
- åããããŒãèªã¿åããŸãã
- ããŒãžããšã«èšé²ãç ŽãïŒãã®äŸã§ã¯10åã®ã³ã¬ã¯ã·ã§ã³ïŒã
- ç®çã®ããŒãžãéžæããŸãã
- åãšã¬ã³ãŒãã®ååããããããŸãã
- çµæãã³ã¬ã¯ã·ã§ã³ã®åœ¢åŒã§æäŸããŸãã
é¢æ°ã³ãŒãèªäœãšæè¡çãªå®è£
ã«å€ãã®æéãè²»ããããšã¯ãããŸãããããã§ã¯ãã¹ãŠãéåžžã«åçŽã§ãããå®å
šãªã³ãŒãã¯ç§ã®GitHubã§å
¥æã§ããŸã ã
for i in range(0, len(lines_proc)): d = dict((u''.join(key), u''.join(value)) for (key, value) in zip(headers, lines_proc[i].split("\t"))) response_body.append(d) return { 'statusCode': 200, 'page' : num_page, 'body': response_body }
ãé¢æ°ã³ãŒãããããã¯ã«ã³ãŒããæ¿å
¥ïŒãŸãã¯ç¬èªã«äœæ:)ïŒããç»é¢ã®å³äžé
ã«ãããä¿åããã¿ã³ãã¯ãªãã¯ããŸãã
- ãã¹ãã€ãã³ããäœæããŸãã ã³ãŒããæ¿å
¥ãããšãé¢æ°ãèµ·åããŠãã¹ãã§ããããã«ãªããŸãã [ãã¹ã]ãã¿ã³ãã¯ãªãã¯ããŠãããã€ãã®ãã¹ãã€ãã³ããäœæããŸããç°ãªããã©ã¡ãŒã¿ãŒã§lambda_handleré¢æ°ãèµ·åããŸãã ããªãã¡ïŒ
- ãã©ã¡ãŒã¿ãŒ 'page'ã§é¢æ°ãéå§ïŒ '100';
- ãã©ã¡ãŒã¿ãŒ 'page'ã§é¢æ°ãèµ·åããïŒ '1000000';
- ãããŒãžããã©ã¡ãŒã¿ãŒã§é¢æ°ãéå§ããïŒãbla-bla-blaã;
- ãããŒãžããã©ã¡ãŒã¿ãªãã§é¢æ°ãå®è¡ããŸãã
ãã¹ãã€ãã³ãPage100
ãã¹ãã€ãã³ãããŒãž== 100ãéä¿¡ããŠãäœæãããé¢æ°ãéå§ããŸãã以äžã®ã¹ã¯ãªãŒã³ã·ã§ãããããããããã«ãé¢æ°ã¯æ£åžžã«æ©èœããã¹ããŒã¿ã¹200ïŒOKïŒãè¿ããŸããã
ãã¹ãã€ãã³ãPage100ã®ããªã¬ãŒ
å®éšã®çŽåºŠãé«ããããã«ãå¥ã®ãã¹ãã€ãã³ããPageBlaBlaBlaããéå§ããŸãã ãã®å Žåãé¢æ°ã¯çµæã«ã³ãŒã415ãšãæž¡ããããã©ã¡ãŒã¿ãŒã®æ£ç¢ºæ§ã確èªããå¿
èŠããããšããã³ã¡ã³ããè¿ããŸãã
ãã¹ãã€ãã³ãPageBlaBlaBla
PageBlaBlaBlaã€ãã³ãã®ããªã¬ãŒ
APIã®äœæ
ä»ã®ãã¹ãŠã®ã±ãŒã¹ããã¹ãããLambdaé¢æ°ãæåŸ
ã©ããã«æ©èœããããšãç解ããããAPIã®äœæã«é²ã¿ãŸãã äžèšã§äœæããLambdaé¢æ°ãžã®ã¢ã¯ã»ã¹ãã€ã³ããäœæããAPIããŒã䜿çšããŠäžèŠãªèµ·åã«å¯Ÿããä¿è·ãè¿œå èšå®ããŸãã
çµè«ãšãŸãšã
ãã®èšäºã§ã¯ãAmazonã¯ã©ãŠããµãŒãã¹ã䜿çšããŠãµãŒããŒãªãã®èªåã¹ã±ãŒã©ãã«REST APIãäœæããæ¹æ³ãæ€èšããŸããã ãã®èšäºã¯æå°ã®ããªã¥ãŒã ã§ã¯ãããŸããã§ããããAPIãäœæããããã»ã¹å
šäœãå¯èœãªéã培åºçã«èª¬æããäžé£ã®ã¢ã¯ã·ã§ã³å
šäœããŸãšããããšããŸããã
ãã®èšäºã§èª¬æãããŠããæé ã1ã2åç¹°ãè¿ããåŸã5å以äžã§ã¯ã©ãŠãAPIãäžããããšãã§ãããšç¢ºä¿¡ããŠããŸãã
AWS API GatewayãµãŒãã¹ã¯ãæ¯èŒçåçŽã§ãå®äŸ¡ã§ã匷åã§ãããããéçºè
ãä»äºãåçšãããžã§ã¯ãã§äœ¿çšããå¹
åºãæ©äŒãéããŸãã ãã®èšäºã®çè«çãªè³æãæŽçããã«ã¯ãç¡æã®Amazon Web Services幎éãµãã¹ã¯ãªãã·ã§ã³ã«ãµã€ã³ã¢ããããäžèšã®æé ãå®è¡ããŠREST APIãèªåã§äœæããŠã¿ãŠãã ããã
ã質åããææ¡ã«ã€ããŠã¯ãåãã§ã話ãããŸãã ãã®èšäºã«å¯Ÿããã³ã¡ã³ãããåŸ
ã¡ããŠãããŸããæåããç¥ãããŸãã